The real mistake started with wanting to finish adding email subscription support on Thursday.
I felt so productive the past week studying, so adding another finished task to the list of small victories while I was on a roll felt like the obvious play.
While the internal testing for the email sending succeeded, the subscription signup form on the home page did not. I kept receiving unknown error. I tried a bunch of things, restarting the daemon, changing up some values in the configuration files, none of them seemed to work.
The final saving throw was to update Ghost and then hope the signup works.
The update started fine. Knex throws a fit every time Ghost updates. The well-documented solution is to do npm install sqlite3 --save --unsafe-perm --sqlite=/usr/local which I do.
This starts the chain of unfortunate of events, because for the first time, it doesn't work. It throws an error.
knex
legitwision
Multiple threads on Github and the Ghost forums complain about @vscode/sqlite3 not being found. Solutions include renaming folders, installing VSCode (???), and changing some package versions around. The main issue is that some time ago, sqlite3 stopped being maintained so knex switched to vscode's fork. According to the devs, since the original sqlite3 package is getting maintainers again, they'll switch package dependencies back to it.
These were threads before the knex and knex-migration, but by the time I was installing it, knex and knex-migration already came out with new updates that switched back to sqlite3. Ghost hadn't updated so I needed to manually change package versions.
Now the problem was figuring out how to properly tell npm to install the new versions of knex. (By now, any thoughts about the damn email signups have been long forgotten.)
Let me spoil the ending and say that all I had to do was to do to update the packages was yarn remove knex and knex-migrator and then repeat with yarn add. Unfortunately I had no clue how to deal with changing package versions and messed around with package.json and npm uninstall and npm install a lot. Smooth brained moment.
When the npm commands didn't work, I resorted to renaming @vscode/sqlite3 to sqlite3 in every package.json for knex and knex-migrator. Tried npm commands again, still not working. After that, I tried manually executing knex-migrator. Still no dice.
After more searching, I find a thread about what to do when software upgrade succeeds but the database migration fails. The suggested solution is to use Ghost's built-in rollback feature. That seemed like a good idea. No thoughts about broken packages, only thoughts about getting the site back up. Forget the email signups.
I try the rollback, only for it to also fail, because knex's database migration failed in the middle. I get an error that the database's migration lock was never released. Unfortunately, that means I need to manually go open the database using sqlite3 and use a command to drop the migration lock. THEN I have to pray that the database wasn't irreversibly corrupted during all the bullshit that the software upgrade attempts have caused it.
KeenenCharlesThe real kicker is that this is the one time I didn't backup the essential files before the upgrade. I've done it every time before this, and didn't bother this time because the Ghost version upgrade is a small bump and unlikely to cause problems. Ohoho what a bad day to be lazy. If the database is corrupted, my entire site would have to be reinstalled. Luckily I did do some site data export from January, so I saved the posts after January as HTML pages in case reinstallation was necessary and I needed to restore post content.
I download sqlite3 and execute UPDATE migrations_lock set locked=0 where lock_key='km01'; as the thread suggests. This is probably the only thing in this whole dumpster fire that goes right, because the database is thankfully uncorrupted and the rollback succeeds. Then I try logging into the admin panel, but somehow the login page throws an "unknown error" and refuses to log me in. Could the database still be corrupted even though the rollback worked? I check the database for users, and my account shows up, with salted password and other metrics. The data is there, but login page refuses to work.
At this point it's probably 2am and it's too late to back out now. If I went to sleep right there, it'd break the flow of everything I've already done and trying to pick it back up in the morning would be less productive. So I continue.
After some more headbanging and StackOverflow, I finally bother to look into how yarn might be able to help. yarn.lock also had references to @vscode/sqlite3 but I didn't change them. npm doesn't interact with yarn, after all. But following some breadcrumbs, I get the hint to try some yarn commands, and what do you know. That was the solution to properly installing newer versions of knex and knex-migrator. npm install commands didn't work, mostly because it refused to install newer versions. yarn had no such restrictions and properly installed the latest versions.
Finally, with the fixed package versions, I try updating Ghost again. The database migration and software upgrade succeeds this time. The time is now 3am and I was sitting there feeling incredibly stupid at how my struggle from 10pm to 3am could've been saved by straightforward yarn usage. This is one expensive lesson I won't forget for sure.
Although I doubt it was the reason, the software upgrade managed to fix the email issue. The login issue was due to a database permissions error. Setting permissions correctly fixed the login, and everything was finally fixed.
Of course, dumping extra salt in the wound, hours later, Ghost pushed out a new software version that took care of all the new knex and knex-migration version requirements instead of requiring manual intervention.
One day. ONE DAY. All I needed to do was wait an extra day on top of the weeks of procrastination already spent on not updating the blog and I wouldn't have wasted my life on maintenance.
life sux
The only positive to come out of this is the notification that Ghost 5.0 will no longer support any database in production except MySQL8. My host only uses MariaDB and it's no longer compatible with MySQL8. Ghost 4.x end-of-life isn't for at least another year, so I have some time to figure something out. Switch to another blogging engine, or switch hosts? Cheap VPS hosts are plentiful now, and tutorials about managing every application under the sun are widely available, so it's not a bad idea to get some new experience. We'll see.