DJ5CW's Ham Radio Stuff

All kinds of things too precious not to share, in short articles. Constantly under construction or destruction. Note that code snippets are usually working minimum examples (often stripped of extra features). If you plan to use and of the code for anything serious, you may want to contact me. It's likely that I can provide you with some updates and good (?) advice. fabian@fkurz.net

Fabian Kurz → Ham Radio → Stuff

ETM-DIGI-1 interface

I recently acquired an old ETM-DIGI-1 interface without documentation. It took a while to find the manual and schematics of this interface, but I eventually found it, thanks to the amazing Internet Archive's Wayback Machine. Here it is (German language only):

• ETM-DIGI-1 Bedienungsanleitung und Schaltplan
• ETM-DIGI-1 Testberichte

Added: 31-Jan-2021. Last modified: 31-Jan-2021.

Morse Distance Calculator

Impatient? Here's the direct link.

On the CW Ops mailing list, Tim/N3QE started a discussion titled Measuring "nearness" of callsigns in Morse Code space?.

[...]
Are there any measures for "code space" difference between callsigns in Morse
code?

For example, "different by only one element" (nothing<->dit<->dash) would be
one possible measure. Most of these would change the length of the callsign but
if the change happened at the beginning or the end then you might not notice
that something was missing.

Another possible measure would be "different by bit elements but same in
length". For example if there's a deep fade in the middle of a "T" then it
sounds like an "I". Or a longer fade in the middle of a "O" could make it sound
like two T's "T T".

Super Check Partial does callsign nearness in alphabet space, not morse-space.
And I'm thinking that measuring the difference in morse-space might be more
relevant.

Any ideas or brainstorming?

Tim N3QE

My little contribution to this is a PHP script which calculates the Levenshtein distance of callsigns in Morse "samples":

> Any ideas or brainstorming?

the Levenshtein distance can be used for this. It's a metric to
measure the difference between two strings.

However applying it directly to callsign letters would only give a
very approximate measure of similarity when we're concerned about
Morse code.

Using Dits and Dahs instead of letters already brings us closer, e.g.

-.- --... ... ...-   K7SV
-.- --... ... ...    K7SS

only has a distance of 1 whereas for example K7SQ has a much greater
distance.

Even better, we break down the code into time samples which would then
give a reasonably small error distance for errors where two dits are
merged into a dash, like K7SV <-> K7AV.

--- - ---   --- --- - - -    - - -   - - - ---    K7SV
--- - ---   --- --- - - -    - ---   - - - ---    K7AV

I put this to the test with a small PHP script which can be found
here: http://fkurz.net/ham/stuff/morsedistance.php?callsign=K7SV

You can enter any callsign and it will be tested against all calls of
the current MASTER.SCP file (38k callsigns). Hovering over a callsign
visualises the difference.

It only shows the callsigns up to a Levenshtein distance of up to 10,
but you will see that really similar calls are usually not found
beyond a distance of 4.

73
Fabian, DJ1YFK

P.S.: The source code is available and linked from the page.

Added: 06-Dec-2018. Last modified: 06-Dec-2018.

Kenwood TS-440S/AT repair notes

For my growing station in Poland (SO5CW), I recently acquired a TS-440S/AT for just 120 EUR, and of course it had some issues, which I am in the process of repairing:

PLL unlock - dots in display (fixed)

This issue is very well known and described in detail e.g. on OZ1BXM's TS-440S repair page. The root cause is that the glue which was poured over some VCOs becomes conductive over time. The glue must be removed.

I completely rebuilt VCO1 (RF unit) and then VCO5 (PLL unit), but I didn't use new parts except for Rs and electrolytic caps; instead I carefully removed the glue from the PCB as good as I could, desoldered and cleaned the parts, and finally (after removing all the remaining glue) rebuilt it (📷1, 📷2, 📷3, 📷4). In the process I accidently destroyed a varicap diode which I replaced with a BB139, as suggested by OZ1BXM, and a switching diode which I replaced with a BA243 (OZ1BXM recommends the BA244 which has superior parameters, but I couldn't obtain one). I also killed L43 in VCO5, which was easy to replace, however.

Once everything was rebuilt, on VCO1 I needed to adjust T23 slightly (due to the changed varicap) and that was it! Re-using the old parts probably saved me from some tuning work.

A few things I remember:

• For rebuilding VCO1, remove the RF Unit PCB completely from the transceiver (bottom side). Unlike the PLL Unit (top side), which only has pluggable connectors, the RF Unit has two coax lines soldered directly onto the PCB. However handling the PCB while connected was not practical and I gave up after a little while.
• Removing the RF Unit from the TS-440 chassis takes a little patience, because it is surrounded by wire busses etc. I was afraid that I'd never get it back in again, but it was easier than expected! The best direction to take it out of the transceiver was to the right (looking at the front of the TRX).
• Check the components you unsolder carefully. As written above, I managed to kill two diodes, which was immediately obvious (broken case), but L43 looked perfect from the outside. At least check Ls and Rs and measure diodes with a multimeter. If you can measure Cs, even better (I didn't).
• To keep track of all the components, I took a A4 sheet and put the desoldered components on it, with the component designator next to it (📷5). When I was done removing all components, I put them back in exactly in the reverse order (after cleaning and straightening the legs).
• I did not remove the shielding can around VCO5. This meant that working on the components sometimes felt a bit like doing a keyhole surgery, but in the end it was fine.
• Be careful with D13 of VCO5, as it has a little ferrite bead on one leg.

My TS-440S at SO5CW

The whole rebuild of VCO1 and VCO5 took about 8h total. I listened to great blues by Dire Straits to make it bearable :-)

TS-440S attenuator (ATT) malfunction (fixed)

The attenuator button (ATT) of the TS-440S/AT didn't show any function. Checking the DC levels around Q2 showed that there was only around 6V present at the emitter instead of the expected 13.3V. The voltage supply is coming from J16 via L115/L112. 13.7V could be measured at those Ls, but not at Q2, although according to the circuit diagram there's nothing in between. It turned out, however, that there's a wire bridge W2 between them, and furthermore this was populated with a resistor (4.7 Ohms) - and this resistor was broken (open).

Replacing W2 with a new 4.7 Ohm resistor fixed the attenuator.

TS-440S RIT/XIT malfunction

Turning the RIT or XIT control results in abrupt changes from -1.2 kHz to +1.2 kHz (full scale values), with nothing in between.

An quick check showed that the RIT potentiometer is actually broken. So far I have not made an attempt to repair/replace it, because it a) takes a lot of effort to remove the front panel completely, so I can reach the PCB with the potentiometer and b) I can simply use SPLIT if I need the RIT functionality.

Links

• OZ1BXM's TS-440S repair page - Detail description of the VCO rebuild and other issues
• TS-440S service manual - indispensable resource
• TS-440S service manual and user manual - high quality scans by PU2SEX

Added: 28-Oct-2018. Last modified: 17-Sep-2021.

Websocket Gateway to a DX Cluster

While modernizing the FOC RBN web based telnet "client", which is nothing more than an iframe that loads a HTML page that consists of not much more than a <pre> tag followed by the telnet output, with an infinite timeout so that you actually see the spots in your browser more or less in real time, I decided it was time to have a look at websockets and to build a real web client that allows using a DX cluster interactively just like with a normal telnet client.

Thanks to my recent infatuation with the programming language Go, I picked its websocket server implementation to minimize the lines of code I have to write myself, and half an hour later, it was done: A full websocket to telnet bridge.

On the Go side, it simply calls a handler for each incoming websocket connection which in turn opens a socket to the DX cluster, and then dispatches the output of the cluster back to the websocket in a separate go routine, while reading user input from the websocket and redirecting it to the cluster.

On the HTML side it's a very simple HTML page that includes a little form with a textarea for the telnet output, an input line and a few lines of JavaScript to do the Websocket magic.

You can build some fancy stuff around it, buttons that trigger certain actions or some more JavaScript code that interprets the spots and does something more useful with them than just displaying them line by line... There are no limits to your imagination.

By the way, using mod_proxy_wstunnel you can easily hide the websocket server behind your Apache server, and also provide a secure websocket if your web server is already configured for it, at no extra cost.

Sources

• ws.go - Gateway sources in Go
• spots_ws.html - Website (working example) with minimum JavaScript/HTML code

Added: 21-Aug-2016. Last modified: 21-Aug-2016.

RBN activity charts

This article is a work in progress - it will describe the technical side of the Reverse Beacon Network (RBN) activity charts available at https://rbn.telegraphy.de/activity/DJ1YFK including source code and the rationale behind the design of the system.

Scope of RBN activity charts

The goal of the RBN activity charts is to get a very quick overview of the activity of a callsign, as reported by the RBN. This includes all modes supported by the RBN receivers, currently this is mainly CW with some Skimmers also supporting RTTY, PSK31 and other digital modes.

A desired representation includes a heat-map style calendar view for at least one year back, where the reported RBN activity per day is mapped to a color intensity, very similar to the commit activity maps over at at github. Activity points are awarded for each hour of the day, and within each hour one point for each band on which the RBN reported a spot, plus the number of continents on which the station was heard during that hour. This gives a hopefully more meaningful picture of the activity than just counting the raw number of spots, which can (with a decent signal and suitable propagation) nowadays easily reach 50 for a single CQ.

Additionally, a band breakdown of the reported activity is desired, to get a quick idea of the activity focus of a certain callsign. To round it off, a graph showing the activity versus the hours of the day gives an interesting insight into typical operating hours, and with a little extra work, this chart can also include the bands on which the activity took place.

Finally, a summary active hours per day, total for the last year, and the number of hours in which the callsign was detected by a skimmer on each of the continents rounds off the information presented.

Examples

The finished "product" can be seen on the URL mentioned above. Just for illustration, you may find some of the following exemplary overviews interesting:

• DJ6ZM - Very active CW OP
• N3JT - Peaks for the CWops CWT events
• W6WX - NCDXF Beacons
• 9A1A - M/M contest station

Raw data and how to process and store it efficiently

The raw aggregated data from the RBN is available at reversebeacon.net in CSV format, one file a day. After unzipping the data, typical file sizes range between about 10 MB (150k spots, week days) and over 150 MB (2.5M spots, contest weekend) for a single day, i. e. more than 1 GB per month, and consequently over 12 GB or something in the range of 50 million spots a year. This really is "big data".

The naive way to handle this kind of data would be to pay a lot of money for a high performance server or rent something at AWS, throw each line into a database with a fancy name, install all the frameworks that currently are trendy, and then for each query to generate an activity chart, search through millions of database rows and let the CPU melt away a bit. "Cloud developers" love this approach.

The smart way, however, is to process the data once, convert it into a suitable format in which the desired data is immediately available, and save it efficiently. That's the way I chose, because I love small and scalable solutions, much more than throwing money and ressources at problems until they disappear (which they often won't).

What do we really need?

For the type of activity report we want to generate, there's a simple data relationship, which is fundamentally different from the "table" view of a CSV file with all the spots in single rows: What we need is a key → value relationship between a callsign, and all data that is associated with this callsign.

Furthermore, in order to generate the kind of report described above, we can do away with a lot of redundancy that is contained in the RBN raw data. If we only like to know if DJ1YFK was heard by the RBN on 20m in the 23:00 UTC hour on January 1st, 2015, we don't need to save 50 spots (each with 50 or so bytes) for that, because it's only a few bits of actual information. It doesn't matter which precise skimmers received the call, the continents are sufficient. The same is true for the precise frequency, SNR, CW speed or any of the other information that's available in a fine grained form in the raw data. In the end, the whole information that is saved for one hour in which a callsign was active, is a bit field with 32 bits as tabulated below.

NB: It started out with 16 bits per hour, but later additional bands were added, so the bitfield was increased to 32 bits, which even leaves some room for future additions.

Data representation

The next question was: How to save this bitfield in an efficient manner, and not only for one single hour, but for any hour and date. And it needs to be saved in a way that makes is very easy not only to retrieve this data, but also to update any particular hour or day without much overhead and moving data around.

After considering and rejecting formats like JSON, the best solution was infinitely simpler and faster: All data was stuffed into an array of bytes, where a certain starting point (in this case January 1st, 2009), was byte 0, and for each hour from that point four bytes were allocated. The appropriate bits in this array are then filled day by day with the new raw data.

The nice thing about this way of representing the data is that you can immediately access any hour without doing a lot of math (calculating the hours from a certain starting point is something that library functions do perfectly), and since all hours are pre-allocated, there is no shifting around when adding new data (possibly in a non-chronological way). A year of this data takes 365 * 24 * 4 = 35040 bytes.

With more than 500k callsigns appearing on the RBN within a year's frame, this is a lot of data, more than 16 GB. If you want to save two consecutive years, it's twice that, and with ten years, it's ten times that. Not practical! Fortunately, the data typically contains a lot of zeroes, simply because the activity of most radio amateurs is not spread evenly over the hours of the day, and most hams are not active every single day. This means you can compress the data very well, which - in my case simply with gzip - results in an average size of just above 200 bytes per callsign, a compression factor of about 160. The penalty in speed and additional CPU load introduced by gzip is a small price to pay for that.

Data storage

With the data representation solved, the next issue was how to store it. In a first version, this was done simply in the filesystem: The data for each callsign was saved in a single file - in order to avoid a single folder with half a million files, instead there were a number of folders with the first two letters of the MD5 hash of the call, over which the files were spread out.

This was very ineffective, with every file, no matter how small it is, taking 4 kB on the file system, making the best compression useless. It's also a nightmare to backup so many small files.

So a "real" database was needed. While this sounds like a perfect use case for one of the many key-value store databases, I took a short cut and used MySQL (because it was there already) with a very simple table that had two fields, the callsign (indexed) and a blob data field for the gzip compressed array described above. Without taking a benchmark against anything else this solution "felt" absolutely fine, with very short query times and a total database size of less than 300 MB, I stopped to look for anything better. No need to over-engineer things!

Now everything was in place, the last thing that was needed was the actual software filling the database with the daily batch of RBN data. With all design questions answered, this was written as a small Perl script that is started by a cron job every night. The source files are listed below.

TODO: Add a description of the RBN live/real stime statistics here.

Web frontend

All data is useless if you cannot present it in a way that is easy to read and at least to some degree pleasant looking. It's also important to keep the site size small, use JavaScript carefully and keep the number of external resources that are included low (or avoid them completely if somehow possible). This philosophy is called common sense and is not very fashionable in modern web design any more.

Loading a typical RBN activity chart (it depends a little on the actual activity represented in it) will take less than 25 kB of HTML. The first time you load a chart, an overhead of 1 kB CSS and regrettably about 200 kB of JavaScript are required. And 775 bytes of PNG images. All together you end up with about 75 kB of compressed data that has to be transferred over the wire, when the browser supports gzip compression (and practically all browsers do).

For the heatmap calender, against my habits, I used a third party JavaScript library, cal-heatmap, which itself uses D3.js. The graphs below the heatmap are homemade with 1999 style tables, but it works well enough, with various browsers and gives a decent user experience also on mobile devices. Everything except the heatmap data is produced by a small PHP script. The heatmap script fetches a JSON file generated by a separate PHP script with the data in a way cal-heatmap can understand. I like clean and "speaking" URLs (/activity/CALLSIGN instead of index.php?action=activity&call=CALLSIGN) which is easy to do with Apache's mod_rewrite.

Embeddable version

If you embed an RBN activity chart in an iframe on another website, this will be detected by a JavaScript and the layout is adjusted to optimize the site for embedding it on another website, for example into your profile on HamQTH.com or QRZ.com. The header size is reduced, buttons for band filtering are removed, and a small "banner" is added at the bottom.

Misc.

Performance

Statistics

Sources

The RBN activity charts are part of the offering at rbn.telegraphy.de, and the souces can be found in the git repository https://git.fkurz.net/dj1yfk/cwclubspotter

Added: 07-Jul-2016. Last modified: 27-Jul-2020.

Fabian Kurz → Ham Radio → Stuff