Learning About Debugging

I was reading a Reddit post on how Guido van Rossum stated he wanted to make Python work faster when I came across a comment chain where somebody joked that you can make any program faster by using fewer print() statements. A reply quipped “How else am I supposed to debug my programs?”. I’m assuming it was a joke for them, but this is certainly a weakness on my part. Since I’m self-taught, I find myself using standards that make sense to me, but would be bad practice for a formally taught individual. In this case, that’s using the print() statement to help me debug by running the program and seeing what I end up with, tweaking that, running it again, and so on.

The Reddit thread includes a comment chain where somebody jokes about using fewer print() statements.

Fortunately, though, somebody in the replies provided a serious answer to the quip: debuggers and logging. Because I use Linux, I’ve gotten quite familiar with log files – any time something doesn’t work, you have to check a log file to see where the problem lies. Without log files, you’re utterly confused; with log files, you’re a quick Internet search away from your answer. However, I never applied logging in my own programs. Firstly, they’re often so small that they’re easy to review without a log file; secondly, I just didn’t know how. It’s still bad practice, though.

Fortunately, one person linked to a really solid explanation of how debuggers work from last year’s PyCon:

A video by Nina Zakharenko explaining debugging in Python

It was so enlightening to watch – she described exactly how I muck around debugging my programs, using print() statements to analyze my output, adjusting it, and so on. Clearly, my bad habit was a common amateur move that happened often enough for her to be motivated to tell people about this debugger. Not only does it look like a good habit to build, the way she explains it makes it just so easy that I don’t see why I never bothered to use it in the first place.

Setbacks and new hurdles

I’d left the pomodoro project in the freezer for quite a while. Last week, I figured I’d pick up on it again, and make some time for that hobby. It’s been challenging to pick it up again, however. KiCad, apparently, has had an update, and currently does not work on my PC. However, the error message that I get does not seem to be a common enough experience to warrant many questions about it online. It might be that it’s too recent for it to become an issue (it was a few days ago); it might be that the problem is unique to my setup; or, alternatively, it might be that the problem is really so trivial that most people have just applied some solution that I do not know.

Regardless, I figured to just work directly on the programming of the new uC, as that is a thing I can experiment with on the go without needing KiCad. Everything works fine for a few iterations as I test the LEDs by blinking them all in order, and I test various waiting functions, and so on. As I was testing out interrupt states, I started getting an odd message – for some reason, Avrdude was setting new fuses. On the next flash, it mentions that fuses have changed, and asks whether I want to reset them. Naturally, I do. However, the fuse setting issue seems to repeat, and then I can’t seem to connect at all anymore.

So, I figure to test some alternatives. Can I upload something else to the chip? Can I use the USBTiny to upload something to my first prototype? Is it the connections on the breadboard, is it the USBTiny PCB itself? What’s going on? All tests fail. So, a new challenge presents itself. I need to figure out where the problem is, followed by what the problem is, and then whether or not this is something that I can fix.

I use Arch, BTW

This week was a holiday for me, fortunately. Technically, my work doesn’t offer standard holidays, except for the usual summer break in university education. However, after the merger with another local university, our new colleagues brought with them a lovely cultural shift: the assumption that the education-free weeks are holidays. So many of them take this as just a right, that our employer has basically given up and accepted this is a holiday week for educational staff. Part of that, I suspect, is the fact that 1 in 5 employees in education suffers from some sort of burn-out, according to the Dutch national statistics centre, CBS (at the top of that first graph, “Onderwijs” means “Education”). I imagine that this certain lax attitude towards the enforcement of certain policies is just considered risk management – the potential damage of formally enforcing this as not being a holiday would likely outweigh the financial benefits of having teaching staff available during this week.

So, I finally had some time and energy on my hands to do some things I’d been looking forward to for a while. One of which was to transfer my desktop PC over to Arch from Manjaro Linux. I’d originally kept my main PC on Manjaro based on the idea that I wanted a PC that just worked without much fuss, and assumed that personally maintaining Arch would cause more issues than having a more curated OS. Ironically, over time, my Manjaro-based desktop ended up having issues more often than my Arch-based laptop. Every so often, an update to Manjaro would prevent me from logging in to the X Server (an update to NVidia drivers would end up conflicting with lightdm-slick-greeter, or reset a configuration file), and I’d be left fixing that. Alternatively, I’d set things up exactly the way I wanted to on my laptop, and when I’d go back to using my desktop, I realized that it had a different setting or program that I did not prefer at all. So, with some more time on my hands, it was time for a change.

As I’d written before, I’ve installed Arch Linux a number of times over the past years, so the installation itself wasn’t that much of a problem. I was, of course, irrationally worried that I’d irreparably mess something up but that’s just usual nerves; the reality is that very little in Linux is irreparable. In fact, the process was surprisingly smooth and easy. Particularly given how Linux separates out files in its system hierarchy, it was incredibly easy to change the OS and still keep my old files and even installed games. So, now I finally fully run Arch Linux as a main driver, and I’m happily configuring and customizing when I want and as I see fit.

Hit My Tax with Assembly

Wow, Assembly is hard! I’ve been following that tutorial I mentioned a while back, and I’ve gotten to the point where I can’t follow along anymore. In lesson 6 about timer interrupts there’s quite a jump to new programming concepts and program structure, without much more guidance. So far, I’ve been able to follow along quite well, and though I wouldn’t be able to easily program something in Assembly just yet, I at least felt comfortable enough to read and understand an Assembly program. At this point, though, it’s becoming quite tricky to follow.

Part of the difficulty, I think, is that it’s clear that the website is originally written in German, and while I’m able to follow the text (fortunately, knowing some German helps me follow the sentence structure), the English translation can be a little rough. It’s terse and clipped prose, packed with information. The tutorial is also written more as a demonstration than a lesson plan. Educationally speaking, I would have put in exercises inbetween or some small challenges at the end of lessons with proposed answers at the back. At least something to provide a little more structure to the whole.

The other part, though, is the massive jump from lesson 5 to 6. The earlier lessons all involved a single pin goal (blinking or dimming it) with only a few registers. In lesson 6, though, suddenly we’re setting multiple registers, running two processes in parallel, on top of learning timer interrupts and the way it completely changes the program structure. Simply said, I’m lost!

I already knew that I wouldn’t end up actually programming much in Assembly – seeing how easy Arduino programming is predisposes me to trying C – but I figured to at least learn the basics so I can understand better what’s happening in C. At this point, the cost of investing time into learning this doesn’t really outweigh the benefit of understanding the workings. I figure that, at least for now, I understand more than I did before, and must rather work on the next project.

My First Makefile

Following that AVR Assembly tutorial I’ve been mentioning recently, I’m finding myself compiling and flashing programs to the ATtiny13 quite frequently. So, after doing that twice, I was done with repeating those steps over and over, when I could just automate that part. My first instinct was to use a simple Shell script to put the commands in, but Linux wouldn’t be Linux if somebody hadn’t already run into and solved that very same issue. So, today I learned about Makefiles.

The way I understand it, make is a program designed help in compiling – it checks whether any source files have changed and, if so, compiles them again. In programs of the size I’m writing, that’s really a negligible advantage, but I can see with larger programs that more than a second to compile having to wait over and over can get tiring quickly. The more interesting part for me is that I can use it to immediately say “flash this to the ATtiny13!” and the makefile will take care of the configuration of the commands to compile and send it. Right now, I have the following file, which I’m sure will expand as I learn about makefiles and how they work:

MICROCONTROLLER = attiny13
PROGRAMMER = usbtiny
INCPATH = /home/nextcloud/lib/avra
SRCFILE = 5_fast_pwm.asm

$(SRCFILE).hex: $(SRCFILE)
        avra -l $(SRCFILE).lst -I $(INCPATH) $(SRCFILE) -o $(SRCFILE).hex

flash: $(SRCFILE).hex
        avrdude -v -p $(MICROCONTROLLER) -c $(PROGRAMMER) -U flash:w:$(SRCFILE).hex:i

clean:
        rm -f $(SRCFILE).hex $(SRCFILE).obj $(SRCFILE).eep.hex $(SRCFILE).asm.lst

Right now, the filenames it creates are a little ugly (“5_fast_pwm.asm.hex”, for instance), so I’m sure I’ll learn how to deal with that better in time. One of my favorite features is the clean command, though. This way of doing it ends up with me having a lot of cluttering files in there, and having one command in the same set to clean it is great.

AVRDUDE Permissions for USB Programmer on Arch Linux

Thanks to a very helpful blog post, I managed to resolve the permissions issue for my USBtiny ISP programmer. As I mentioned in my previous post, I had a permissions error when using it with AVRDUDE, and I wasn’t sure where or how to resolve the issue. As it turns out, there’s a standard config folder where I could just edit a simple file that resolves everything.

Firstly, I needed to see what the vendor and product ID’s of the USB device where. That’s easy to get in Linux with the following command:

[paulo@deku ~]$ lsusb
. . .
Bus 002 Device 004: ID 1781:0c9f Multiple Vendors USBtiny
. . . 

That ID lists the vendor:product IDs. Once you have those, browse on over to /etc/udev/rules.d to make a file called 99-usbtiny.rules, and enter the following:

# Set Group for USBtiny
SUBSYSTEM=="usb", ATTRS{idVendor}=="1781", ATTRS{idProduct}=="0c9f", GROUP="uucp"

As long as you make sure that your user is a member of the group uucp, then you can now use the programmer without a hitch. As the original blog post lists, I have to note that uucp is a group for Arch Linux, and that this will no doubt be different for other distros. Either way, I’m happy to have another issue resolves, and another lesson learned.

Assemly Programming on Linux

As I mentioned a few posts back, I’ve been interested in following a tutorial on programming microcontrollers in Assembly. There’s been some interesting challenges to overcome, but this morning I’ve finally done it and can actually start the tutorial proper. Considering how tricky it was for an average schmuck like me to do this, I figured it’d be worth a small write-up for other people wanting to try this as well.

Firstly, I noticed that the originating pool of information on Assembly programming online now has a surprisingly small number of sources. Mostly, if you’ll look into the uses of AVRDUDE to upload programming to microcontrollers, it will be about uploading C code. The same goes for the main Arch Wiki article on compiling for AVR. That makes perfect sense, of course, because it’s easier to do more with C than with Assembly for the average person; and, from what I’ve read, the speed advantages of Assembly over C for a lot of these purposes are negligable. Apparently, AVR microchips are already optimized quite a bit for C, and the slight advantage that you may get by programming in Assembly pales in comparison to just getting a faster microchip to begin with. So, given that programming these in Assembly mostly seems to be a niche hobby (the programming equivalent of choosing off-the-grid survival rather than camping, perhaps?), there seem to be less than a dozen people online that are a solid source for programming in Assembly, and less when you want answers directly for Linux.

The issue there is that one of the main methods of doing Assembly coding and uploading seems to be Microchip Studio (formerly Atmel Studio, and often referred to as such online, still). The big issue being that this is online available for Windows. So, for Linux, you have to get a little creative. The actual programming, of course, isn’t that tough: use whatever editor or IDE you prefer; my favorite is still Vim, largely because my needs are small and this seems quite fine. The first problem is compiling your assembly, however.

As wonderful as the Arch Wiki usually is, the only relevant post I could find about Assembly is their listing of programming languages. All that tells me is that as is the compiler provided by the packages binutils, fasm, nasm, and yasm. Those, though, are general Assembly compilers. What it doesn’t mention is that there’s also a compiler available specified for the Atmel AVR microchips: avra. Furthermore, the person who wrote the tutorial I linked to at the start also coded his own assembler: gavrasm. Once you’ve written your Assembly file, compiling is ridiculously easy. For example, using gavrasm, you just type:

gavrasm -seb example_file.asm

Or for avra, it’s:

avra example_file.asm -o example_file.hex

Both will give you a hex file that you can upload to your microcontroller. A crucial difference in my case with the ATtiny13 is that gavrasm had its own definitions file for AVR microcontrollers, so didn’t need me to include those. Avra, however, doesn’t, and needed me to supply it. Fortunately, somebody has kindly provided a whole list of them over on GitHub, so I just downloaded the relevant one for me, which turned out to be “tn13def.inc”.

After compiling the .hex file, I needed to figure out how to upload it to the microcontroller via ISP. The tutorial linked above explained a good setup for an ISP header set, and the right connections to the microchip in the first lesson, but it used Microchip Studio for the uploading. So, I had to figure out the Linux alternative. AVRDUDE is that solution, but it took a little more fiddling to get working. Firstly, I needed to figure out what exactly my usb programmer was called (as there are multiple types). Using the command “lsusb” gave me the following output:

Bus 002 Device 013: ID 1781:0c9f Multiple Vendors USBtiny

So, I guessed that “usbtiny” would be the correct name. Using the information provided by Mitchel Davis in episode 5 of his YouTube playlist on Bare-Metal MCU, I ended up with the following command for AVRDUDE:

avrdude -v -p attiny13 -c usbtiny -U flash:w:2_led_on.hex:i

That calls AVRDUDE in verbose (-v) mode for more feedback, telling it we’re programming an attiny13 (-p attiny13) using a usbtiny programmer (-c usbtiny), and that we’re uploading and writing to flash memory, the hex file (-U flash:w:2_led_on.hex) and doing that immediately (:i). That looked great, except it didn’t work. Luckily, the verbose command let me know that it was a permissions issue for the USB programmer. So, sudo-ing the command made it work. Ideally, I’d like to find the correct permissions setting for my user so I wouldn’t have to use sudo for this, but I haven’t been able to find the right one yet. Neither adding uucp nor tty has helped, nor has installing usbasp-udev and avrisp-udev.

So, there’s more to learn still. Possibilities to improve the workflow, for instance, lie in using makefiles to set up some standard settings. And, of course, trying to fix those permissions errors. Either way, with the process described here, I managed to write, compile, and upload the relevant program from the tutorial linked above. So, more learning lies ahead!

Arch Linux and Regular People

Often, when I read about Arch Linux online, there is an odd sense of gatekeeping in comment threads and articles. Arch Linux is seen as a hardcore OS to use. In fact, a common post on /r/archlinux is a proud post describing how somebody spent several hours, trying multiple times to install Arch until they finally managed to make it work. Hardcore fans disparage users of Manjaro as though using Manjaro is a display of weakness. It’s gotten to the point where “btw I use Arch” has become an Internet meme. And yet, I would really recommend Arch.

It’s easy to think that Arch Linux is for programmers; many of the posters online are, or at the least work in IT or have studied Computer Science, or something similar. I am no such thing. In fact, Linux use has taught me some basic programming skills. A common point of praise for Arch is the ability to tweak the system completely to your liking. For some, that means squeezing the absolute most out of their machines: I once read a post where people almost competed to have the quickest boot-up times, arguing over miliseconds as though it was a speedskating event. For me, it’s more about knowing what’s on my laptop. I enjoy the experience of wanting to do something, then just immediately installing a program to do it, and then to go and do that thing. What’s on my laptop is what I put on there, because I wanted it to be.

Doing things this way has taught me about computers, however. Having to learn to program taught me a little about how computers work. Having to understand the Linux filesystems has taught me about the implicit choices in filesystem hierarchies. Having to learn about user and group permissions have made me consider sharing options in Windows as well. Basically, using Linux has made me think about things that I never bothered to think about, even though it’s good to have done so.

I find I reap the benefits of this at work. Now, when we have to switch to a new system, such as when we had to make full use of Office365 when the COVID-19 lockdowns hit last March, I found it easy to adjust to new systems, because I was used to analysing how they interacted. When we needed to set up new interactions between these (such as, let’s say, Microsoft Planner and Microsoft Lists), I could see some analogues to Linux systems, and I could recognize a type of WYSIWYG-interface that tried to abstract code in the background. Despite not working in an IT environment, understanding just a little more of how these systems work has helped me bridge gaps there that I couldn’t have before.

And, on top of that, using Linux is just fun. When I’m back to using Windows 10 on my day-to-day work laptop, it surprises me how clunky Windows is as a system, and how I never noticed that before I really started optimizing my workflow on Linux using i3WM. Let alone that I never bothered to think about workflow before doing that. So, even for us non-programmer, non-IT, non-computery people, I really recommend using Linux.

Experiments in Arch Linux

For the past couple of years, every so often I try to install Arch Linux on one of my laptops to see if I can reasonably work with it. Over time, every time I’ve installed it, I’ve made it more and more functional. Usually, I would into some practical problem that would prevent me from doing what I’d want with it, and I’d end up installing Manjaro on it to just have a working system. The first time, I got stuck on trying to get networking to work (NetworkManager or iproute2 are the answer); the next, I was stuck in the commandline interface (installing xorg-server ended up being the answer); after that, I got an Xorg server running, but couldn’t properly install a desktop environment (I ended up going to i3wm); and so on, and so on. This time, however, I’ve gotten farther than I ever have.

Ever since I fully switched my desktop over to Linux, I’ve had to solve more and more issues myself. Pretty much every time, the real issue was just my ignorance – I’d have updated something without properly reading what the consequences where, or something would break because I didn’t do a simple fix, or things along those lines. The consequence of solving these, though, is that this time around when I installed Arch Linux, I didn’t really run into many issues anymore. Or, more accurately, I encountered the same thresholds, but I’m not getting stuck on them anymore. Sure, I had no sound after the first boot, but that just made sense, because I had no drivers installed. So, a quick install later, some referencing of the Arch Wiki, and I was done.

Similarly, this time around, when I ran into features that I was lacking, I at least had some idea of what I’d want. I’ve experimented a little bit, so I know a program or two that does what I need, and I can more accurately decide which of those I want. Or, as I have been noticing now, I have obtained decent enough skills to implement a minimal fix myself. Running i3wm from a basic install meant I didn’t have my media buttons active yet. No big problem, I just needed to reference the i3wm documentation to see what the syntax was, reference my keyboard layout to see what my key names are (just to confirm it was the default), and them implement a simple volume-up and volume-down key combo. Then I noticed there was no feedback to using those. Well, install a notification daemon, use pactl to list a bunch of info that includes my current volume, grep out the volume, awk out the exact number, pass that to dunst, and done.

Implementing that made me really feel comfortable with Linux. That was a moment when I really saw a set of separate skills come together into a single solution to a simple problem that just a year ago would have me reinstall Manjaro Linux rather than work it out myself. This is the point where there’s a Souls-like enjoyment: I’ve butted my head against the problems long enough, and thanks to my perseverance, I’ve now gained enough experience to start netting a win here and there.

Now, on to solving the next little thing I want to improve in this setup . . .

Excel is not a database

At work, I constantly see people use Excel for things other than spreadsheets. In fact, it’s main use at my job is as a database program; there’s lists of students, lists of grades, group lists, staff addresses, and even tables of standard e-mail responses. One annual task of the committee I chair is that we formally appoint staff members as examiners (giving them the legal right to issue grades to students), and—I kid you not—the standard format for that is a Word .docx file. A list in Word of over 150 colleagues, listing a limited and standardized set of data.

This habit, in my opinion, is a typical example of the saying “to a person with a hammer, every problem looks like a nail.” In this case, people know Excel, and have used it before, so that’s what they use to keep these lists. Now, in the previous academies I’ve worked for, this approach was serviceable. Don’t get me wrong: you can absolutely make Excel do this for you. You can adjust formats, and with a little coding or clever use of Excel validation and HLOOKUP/VLOOKUP functions you can get quite a bit done. However, I see many of my colleagues who have limited knowledge of Excel (and why should they? Often, they don’t need anything beyond a table), so without using this workarounds to have Excel function as a database, I frequently see lots of copy-pasting, manual re-entry of data, or manual formatting to reach the desired result. I can’t tell you how often I’ve seen people copy pasting Excel data from one workbook to another, so that they can manually reformat the data to fit something that they wanted to print.

I myself have worked like this as well. When I just started working as an English lecturer during my MA degree, I saw that my fellow lecturers were all sending grades in a different way to the coordinator. One was making an Excel list using period a comma for the decimal marker (the correct way in Dutch number formatting), while others used commas periods. Yet another just mailed a little list to the coordinator, based on the grades they’d scribbled down on a piece of paper. You can imagine how many mistakes would be made transcribing one to the other, not to mention the work the coordinator had manually reformatting this. So, I made a simple spreadsheet that everybody could access, which had some basic data validation to ensure everybody filled in their grades in the same way. It also solved another issue, because it required everybody to fill in the points students scored, so we weren’t having the usual issue where people rounded grades differently, or changed the way they applied guess corrections for multiple choice, and so on.

Swiftly, this spreadsheet started being used for more things. Random colors started appearing as people marked status changes for students, and comments would start appearing after columns. Also, the spreadsheat kept breaking—colleagues would want to manually adjust this or that, because “it wasn’t calculating the average correctly”, or they would insert columns or new students (despite the groups being generated from those students registered correctly). In short, I learned through experience what people trained in software development already know: your design works until its first encounter with a user. So, I learned about locking cells and password protecting sheets. The last important lesson I learned from that sheet, was that I couldn’t assume people were willing to learn. I’d set it up with the intention that this was a relatively simple sheet, that with around thirty minutes of looking it through could be used and maintained by anybody; after all, it was just a little Excel sheet—not so difficult, right? That’s when I learned that making the sheet had made me into the admin of it, year after year. My colleagues weren’t particularly interested in learning how to better use the programs they were using, because they could already get the output they needed from it. Plus, the urgent matters of the day were so distracting, that they didn’t feel like they could spend the time to learn a new skill, despite that new skill potentially saving them hours and hours down the line.

Now, in some of my old positions, using Excel in ways it was not designed to be used didn’t cause that many issues (nor does it do to this day). A person typing up a quick list in Excel still reaches their end goal. An administrative support worker copy-pasting things from one list to another to edit something doesn’t spend too much time if the whole program has only around 500 students. My new academy, however, has over 3000 students, and when I review the Excel files that people use as databases, it’s just slow. Scrolling down it takes time, and with that number of students, it’s also not handy trying to filter out certain things.

So, fortunately, being the Chair of my committee (and a stubborn one at that), I’m moving us over to an actual database for registration. Much like Excel, when I was a starting lecturer, I don’t actually know how to use Access right now. To me, that’s a good thing—I just love learning, and this is a good opportunity to learn a new thing that will have a practical and positive effect on work. I’m also excited to implement this change, and to help my colleagues save time. I’m convinced this will make processes move more quickly, and I can’t wait to measure the improvement.