All kinds of things too precious not to share, in short articles. Constantly under construction or destruction. Note that code snippets are usually working minimum examples (often stripped of extra features). If you plan to use and of the code for anything serious, you may want to contact me. It's likely that I can provide you with some updates and good (?) advice. email@example.com
Fabian Kurz → Ham Radio → Stuff
I recently acquired an old ETM-DIGI-1 interface without documentation. It took a while to find the manual and schematics of this interface, but I eventually found it, thanks to the amazing Internet Archive's Wayback Machine. Here it is (German language only):
Added: 31-Jan-2021. Last modified: 31-Jan-2021.
Impatient? Here's the direct link.
On the CW Ops mailing list, Tim/N3QE started a discussion titled Measuring "nearness" of callsigns in Morse Code space?.
[...] Are there any measures for "code space" difference between callsigns in Morse code? For example, "different by only one element" (nothing<->dit<->dash) would be one possible measure. Most of these would change the length of the callsign but if the change happened at the beginning or the end then you might not notice that something was missing. Another possible measure would be "different by bit elements but same in length". For example if there's a deep fade in the middle of a "T" then it sounds like an "I". Or a longer fade in the middle of a "O" could make it sound like two T's "T T". Super Check Partial does callsign nearness in alphabet space, not morse-space. And I'm thinking that measuring the difference in morse-space might be more relevant. Any ideas or brainstorming? Tim N3QE
My little contribution to this is a PHP script which calculates the Levenshtein distance of callsigns in Morse "samples":
> Any ideas or brainstorming? the Levenshtein distance can be used for this. It's a metric to measure the difference between two strings. However applying it directly to callsign letters would only give a very approximate measure of similarity when we're concerned about Morse code. Using Dits and Dahs instead of letters already brings us closer, e.g. -.- --... ... ...- K7SV -.- --... ... ... K7SS only has a distance of 1 whereas for example K7SQ has a much greater distance. Even better, we break down the code into time samples which would then give a reasonably small error distance for errors where two dits are merged into a dash, like K7SV <-> K7AV. --- - --- --- --- - - - - - - - - - --- K7SV --- - --- --- --- - - - - --- - - - --- K7AV I put this to the test with a small PHP script which can be found here: http://fkurz.net/ham/stuff/morsedistance.php?callsign=K7SV You can enter any callsign and it will be tested against all calls of the current MASTER.SCP file (38k callsigns). Hovering over a callsign visualises the difference. It only shows the callsigns up to a Levenshtein distance of up to 10, but you will see that really similar calls are usually not found beyond a distance of 4. 73 Fabian, DJ1YFK P.S.: The source code is available and linked from the page.
Added: 06-Dec-2018. Last modified: 06-Dec-2018.
For my growing station in Poland (SO5CW), I recently acquired a TS-440S/AT for just 120 EUR, and of course it had some issues, which I am in the process of repairing:
This issue is very well known and described in detail e.g. on OZ1BXM's TS-440S repair page. The root cause is that the glue which was poured over some VCOs becomes conductive over time. The glue must be removed.
I completely rebuilt VCO1 (RF unit) and then VCO5 (PLL unit), but I didn't use new parts except for Rs and electrolytic caps; instead I carefully removed the glue from the PCB as good as I could, desoldered and cleaned the parts, and finally (after removing all the remaining glue) rebuilt it (📷1, 📷2, 📷3, 📷4). In the process I accidently destroyed a varicap diode which I replaced with a BB139, as suggested by OZ1BXM, and a switching diode which I replaced with a BA243 (OZ1BXM recommends the BA244 which has superior parameters, but I couldn't obtain one). I also killed L43 in VCO5, which was easy to replace, however.
Once everything was rebuilt, on VCO1 I needed to adjust T23 slightly (due to the changed varicap) and that was it! Re-using the old parts probably saved me from some tuning work.
A few things I remember:
The whole rebuild of VCO1 and VCO5 took about 8h total. I listened to great blues by Dire Straits to make it bearable :-)
The attenuator button (ATT) of the TS-440S/AT didn't show any function. Checking the DC levels around Q2 showed that there was only around 6V present at the emitter instead of the expected 13.3V. The voltage supply is coming from J16 via L115/L112. 13.7V could be measured at those Ls, but not at Q2, although according to the circuit diagram there's nothing in between. It turned out, however, that there's a wire bridge W2 between them, and furthermore this was populated with a resistor (4.7 Ohms) - and this resistor was broken (open).
Replacing W2 with a new 4.7 Ohm resistor fixed the attenuator.
Turning the RIT or XIT control results in abrupt changes from -1.2 kHz to +1.2 kHz (full scale values), with nothing in between.
An quick check showed that the RIT potentiometer is actually broken. So far I have not made an attempt to repair/replace it, because it a) takes a lot of effort to remove the front panel completely, so I can reach the PCB with the potentiometer and b) I can simply use SPLIT if I need the RIT functionality.
Added: 28-Oct-2018. Last modified: 17-Sep-2021.
While modernizing the FOC RBN web based telnet "client", which is nothing more than an iframe that loads a HTML page that consists of not much more than a <pre> tag followed by the telnet output, with an infinite timeout so that you actually see the spots in your browser more or less in real time, I decided it was time to have a look at websockets and to build a real web client that allows using a DX cluster interactively just like with a normal telnet client.
Thanks to my recent infatuation with the programming language Go, I picked its websocket server implementation to minimize the lines of code I have to write myself, and half an hour later, it was done: A full websocket to telnet bridge.
On the Go side, it simply calls a handler for each incoming websocket connection which in turn opens a socket to the DX cluster, and then dispatches the output of the cluster back to the websocket in a separate go routine, while reading user input from the websocket and redirecting it to the cluster.
By the way, using
mod_proxy_wstunnel you can easily hide the websocket server behind your Apache server, and also provide a secure websocket if your web server is already configured for it, at no extra cost.
Added: 21-Aug-2016. Last modified: 21-Aug-2016.
This article is a work in progress - it will describe the technical side of the Reverse Beacon Network (RBN) activity charts available at https://rbn.telegraphy.de/activity/DJ1YFK including source code and the rationale behind the design of the system.
The goal of the RBN activity charts is to get a very quick overview of the activity of a callsign, as reported by the RBN. This includes all modes supported by the RBN receivers, currently this is mainly CW with some Skimmers also supporting RTTY, PSK31 and other digital modes.
A desired representation includes a heat-map style calendar view for at least one year back, where the reported RBN activity per day is mapped to a color intensity, very similar to the commit activity maps over at at github. Activity points are awarded for each hour of the day, and within each hour one point for each band on which the RBN reported a spot, plus the number of continents on which the station was heard during that hour. This gives a hopefully more meaningful picture of the activity than just counting the raw number of spots, which can (with a decent signal and suitable propagation) nowadays easily reach 50 for a single CQ.
Additionally, a band breakdown of the reported activity is desired, to get a quick idea of the activity focus of a certain callsign. To round it off, a graph showing the activity versus the hours of the day gives an interesting insight into typical operating hours, and with a little extra work, this chart can also include the bands on which the activity took place.
Finally, a summary active hours per day, total for the last year, and the number of hours in which the callsign was detected by a skimmer on each of the continents rounds off the information presented.
The finished "product" can be seen on the URL mentioned above. Just for illustration, you may find some of the following exemplary overviews interesting:
The raw aggregated data from the RBN is available at reversebeacon.net in CSV format, one file a day. After unzipping the data, typical file sizes range between about 10 MB (150k spots, week days) and over 150 MB (2.5M spots, contest weekend) for a single day, i. e. more than 1 GB per month, and consequently over 12 GB or something in the range of 50 million spots a year. This really is "big data".
The naive way to handle this kind of data would be to pay a lot of money for a high performance server or rent something at AWS, throw each line into a database with a fancy name, install all the frameworks that currently are trendy, and then for each query to generate an activity chart, search through millions of database rows and let the CPU melt away a bit. "Cloud developers" love this approach.
The smart way, however, is to process the data once, convert it into a suitable format in which the desired data is immediately available, and save it efficiently. That's the way I chose, because I love small and scalable solutions, much more than throwing money and ressources at problems until they disappear (which they often won't).
For the type of activity report we want to generate, there's a simple data relationship, which is fundamentally different from the "table" view of a CSV file with all the spots in single rows: What we need is a key → value relationship between a callsign, and all data that is associated with this callsign.
Furthermore, in order to generate the kind of report described above, we can do away with a lot of redundancy that is contained in the RBN raw data. If we only like to know if DJ1YFK was heard by the RBN on 20m in the 23:00 UTC hour on January 1st, 2015, we don't need to save 50 spots (each with 50 or so bytes) for that, because it's only a few bits of actual information. It doesn't matter which precise skimmers received the call, the continents are sufficient. The same is true for the precise frequency, SNR, CW speed or any of the other information that's available in a fine grained form in the raw data. In the end, the whole information that is saved for one hour in which a callsign was active, is a bit field with 32 bits as tabulated below.
NB: It started out with 16 bits per hour, but later additional bands were added, so the bitfield was increased to 32 bits, which even leaves some room for future additions.
The next question was: How to save this bitfield in an efficient manner, and not only for one single hour, but for any hour and date. And it needs to be saved in a way that makes is very easy not only to retrieve this data, but also to update any particular hour or day without much overhead and moving data around.
After considering and rejecting formats like JSON, the best solution was infinitely simpler and faster: All data was stuffed into an array of bytes, where a certain starting point (in this case January 1st, 2009), was byte 0, and for each hour from that point four bytes were allocated. The appropriate bits in this array are then filled day by day with the new raw data.
The nice thing about this way of representing the data is that you can immediately access any hour without doing a lot of math (calculating the hours from a certain starting point is something that library functions do perfectly), and since all hours are pre-allocated, there is no shifting around when adding new data (possibly in a non-chronological way). A year of this data takes 365 * 24 * 4 = 35040 bytes.
With more than 500k callsigns appearing on the RBN within a year's frame, this is a lot of data, more than 16 GB. If you want to save two consecutive years, it's twice that, and with ten years, it's ten times that. Not practical! Fortunately, the data typically contains a lot of zeroes, simply because the activity of most radio amateurs is not spread evenly over the hours of the day, and most hams are not active every single day. This means you can compress the data very well, which - in my case simply with gzip - results in an average size of just above 200 bytes per callsign, a compression factor of about 160. The penalty in speed and additional CPU load introduced by gzip is a small price to pay for that.
With the data representation solved, the next issue was how to store it. In a first version, this was done simply in the filesystem: The data for each callsign was saved in a single file - in order to avoid a single folder with half a million files, instead there were a number of folders with the first two letters of the MD5 hash of the call, over which the files were spread out.
This was very ineffective, with every file, no matter how small it is, taking 4 kB on the file system, making the best compression useless. It's also a nightmare to backup so many small files.
So a "real" database was needed. While this sounds like a perfect use case for one of the many key-value store databases, I took a short cut and used MySQL (because it was there already) with a very simple table that had two fields, the callsign (indexed) and a blob data field for the gzip compressed array described above. Without taking a benchmark against anything else this solution "felt" absolutely fine, with very short query times and a total database size of less than 300 MB, I stopped to look for anything better. No need to over-engineer things!
Now everything was in place, the last thing that was needed was the actual software filling the database with the daily batch of RBN data. With all design questions answered, this was written as a small Perl script that is started by a cron job every night. The source files are listed below.
TODO: Add a description of the RBN live/real stime statistics here.
The RBN activity charts are part of the offering at rbn.telegraphy.de, and the souces can be found in the git repository https://git.fkurz.net/dj1yfk/cwclubspotter.
Added: 07-Jul-2016. Last modified: 27-Jul-2020.
Fabian Kurz → Ham Radio → Stuff