New blog post is up: how I used Atlas to migrate schemas between different database instances. I discuss my use case a bit and give a concise, useful example.
https://medium.com/@a.j.longchamps/migrating-database-schemas-with-atlas-bd83b969a01d
#blogging #opensource #atlas #atlasgo #database #migration #fastapi #postgres #programming #dba #schema #writing #techwriting #rackroot
I've been able to work on Rack Root some more lately, and been spending a lot of time writing tests for the backend. At some point, I'm going to need to make sure I'm covering ALL the code paths, especially the ones that raise exceptions.
I didn't quite do test-driven development, but I do have most branches of the logic covered for a lot of scenarios. I'm up to 67 tests so far, but I'm sure I'll be over 100 once I add more functionality like associating IPs to devices.
For now though, I'm finally getting to write some of the frontend code to handle DHCP ranges on a given subnet. It does feel good to get back into that code.
#programming #rackroot #python #pytest #opensource #fastapi #vuetify
Search results are working now, all the way through to the front end!
I was curious if I needed to worry about HTTP encoding/decoding of things like spaces/%20, but it looks like everything from the frontend to the backend is helping me with all of that. Search results are coming back as expected, though notably not partial string matches. To find the string DS414 for example, you need the whole DS414 and not just 'DS', which isn't the behavior I want. That'll be the next thing to fix.
I am also going to need to fix how the backend is talking to the database, which may mean fixing all of the fields to be lowercase/snake_case instead of camelCase. I discovered that when I was working on the triggers for finding these search results. My preference would be to name the database field somewhere in the Column(...) definition instead of refactoring most of my API calls, but we'll see what SQLAlchemy has for me there.
Then I'll add search support for networking/DHCP and I can get past this hurdle.
#http #programming #vuetify #fastapi #postgres #sqlalchemy #rackroot #homelab
Tests are passing?! I can go to bed. Yay.
Mini Rack Root update: I've been working on search functionality, because I don't have enough in the backlog already for Rack Root, and I've discovered just how little I know about database searches. After MANY iterations back and forth on all the ways not to do it, I've finally found a working piece of code.
I even used ChatGPT, which I almost never do and it was mostly helpful. Sometimes a search engine won't quite get you there and that's how ChatGPT helped. Here's an example with code, help me fix it.
I've also had oh-so-much fun with database schemas being case sensitive (camel case is out, snake case is in), triggers going...somewhere unknown, calling make_searchable() in the right places, and making sure my test environment was _really_ doing what it should have when I dropped the tables to recreate everything. Testing is close to, but not the same as, day to day use when an app is running, of course.
That's a really good stopping point for tonight and I will take it. 55 tests, all green.
#homelab #programming #rackroot #opensource #sqlalchemy #python #postgresql #alwayslearning
Rack Root update: tonight I finally got the search page working and passing parameters the way I need it to. There's a search box in the top right hand corner that will take in a given string and (eventually) go ask the API for objects that match the search terms.
This took a while to put together since entering a search term will redirect you to `/search?q=foo` and that renders properly the first time. With Vuetify however, if you enter another search term while on that page, the app sees you're already on `/search` and doesn't update the values on the page, even though the search parameter is different.
The solution was thankfully found in a GitHub issue (linked below) where other people were looking for the same kind of functionality. This also taught me about the Navigation Guards that Vue/Vuetify come with to handle page navigation/movements. The app knows when you're on the same page and will make decisions based on that.
During my testing I just echoed the search parameter back to the user, but now that the hard part is done, I just need to write the API for it and render that on the frontend. Surely there won't be any complications with asking the API for multiple different objects back... that'll really test my database and python skills.
https://github.com/vuejs/router/issues/1257#issuecomment-2072052011
#rackroot #programming #homelab #justkeeplearning #vuetify #webdev #developer #fastapi #python
It's been a while since I've posted an update for Rack Root, but I've actually made some progress on it lately.
I've finished my refactor to use SQLAlchemy for the database connections and also (finally) figured out how I want to join tables together and relate things.
I also changed some API endpoints. For example, if you look for /network/$id/gateway and there's no gateway, you get a 404. I also changed the UX around gateways. When you make a new network, there's no gateway assigned, so that part of the page has a green + icon. When you set a gateway, that icon is now a red delete icon. Those update the field and icon dynamically without having to refresh the whole page.
The new networks page also will look at the result and see if you got an HTTP 201. If so, you get redirected to the new page. Else, you get an error message. I'm not sure if I want to add more detail to that, it would require the frontend to parse and guess what went wrong. Maybe that's an assignment for another day.
Finally, there's a new Delete network button on the network detail page.
#rackroot #homelab #vuetify #fastapi #sqlalchemy #frontend #backend #fullstack #pytest
I think I finally have my SQLModel to SQLAlchemy refactor completed for Rack Root. As of now, I at least have all of my tests passing and typos worked out. I know I need to add some more join statements, but I'll do those as I come across them.
I still have a little bit of cleanup to do in removing commented code, fixing comments, organizing imports, and syncing this back to the main branch, but the hard part is done.
Then I can finally get back to the front end work. I've been on the backend side of things for what feels like months.
#sqlalchemy #fastapi #webdev #backend #databases #opensource #rackroot
I'm back in the code for Rack Root tonight and realized why I have some of the database relationships I was thinking of getting rid of yesterday. I need ways to track allocations of IP addresses in DHCP ranges and the way I have it setup, there's a nullable foreign key from an IP record over to a DHCP range.
The code for allocating/deallocating an IP in there is easy - if there's a FK set, then you can't set another one.
This is also where I really wish there were better examples of ForeignKey and ForeignKeyConstraints in SQLModel. So far, my searching hasn't turned anything up which might be an opportunity for a blog post and/or pull request.
I need to learn more about how databases work and the best way to setup relationships in SQLModel. I was working on Rack Root tonight and am still running into an issue with some circular dependencies I'm having between various tables. I need these relationships to enhance the data/utility of the app, but might need to rethink how I'm actually putting them together.
The other fun part of this is that I'm learning that the way I test the app and the way I run the app isn't always the same. The tests will run fine the first time against my test Postgres container, but subsequent runs don't work because of how I tell the test DB to drop all data before running the tests. It's complaining about the relationships, but really I think I need to tell it to either cascade delete or just ignore something.
#homelab #rackroot #programming #webdev #fastapi #sqlmodel #databases #alwayslearning
New blog post is up!
How I handled tracking DHCP ranges in Rack Root, some things I ran into along the way, and how I improved my pytest output.
https://medium.com/@a.j.longchamps/rack-root-devlog-tracking-dhcp-ranges-c8272001fe3a
#pytest #homelab #rackroot #devlog #blogging #fastapi #backend #databases #sqlmodel #programming #blogging
After roughly six months of development, I finally put together a blog post introducing Rack Root over on my Medium blog. I go over my inspiration for the project, the major components, and how I keep development sustainable.
#rackroot #homelab #webdev #programming #frontend #backend #python #opensource #portfolio
Turns out, updating the front end of Rack Root was way faster than the backend, this time around.
Tonight, I fixed the API calls for the network details page and also added summary counts for IPAM utilization. I decided I was most interested in how many IPs were available and how many were allocated to DHCP.
I'm not sure if Vuetify has a visualization I can use for this, but the closest thing to what I'm looking for is a horizontal bar with different segments for each type, sized to the percentages of each kind of IP status. If I have 50 DHCP IPs on a /24 network, then roughly 20% of the bar should be colorized to reflect that.
After that, I'm not sure if I'll tackle the DHCP side yet or start capturing details on associated NICs from a device. Either way, that's a whole bunch more relationships to plan out in the database. And now that I have a working example of foreign key relationships, it should be easier.
#rackroot #programming #webdev #frontend #backend #vuetify #fastapi #homelab
Tonight's dev log update for Rack Root
I finally finished my refactor from sqlalchemy with sqlite over to SQLModel with PostgreSQL and refactored much of my code. I had a mixture of snake_cake and camelCase which has since been fixed to be all camelCase.
I adjusted many of my imports, classes, database tables, database foreign key relationships, and testing to fit the new design. I also had to adjust some of my tests because my IDs were off by one.
This database focused refactor also gave me all the structure and space I needed to build out the basic IPAM features. When a new network is added, all the associated IP records are created, and marked as 'Available'. If a gateway is set, that gets changed to 'Gateway' and when I implement DHCP ranges, that will be set to 'DHCP'.
I still have some things to do such as make sure the IP records cascade delete when a network is deleted, and fun stuff like that.
Now that the backend is (re)designed, I can get back to the front end side of things.
If you're curious to check out my code for this refactor, it's all up on my GitHub here:
https://github.com/alongchamps/rack-root/commit/0fc99dfcf10bf3ab6a5dfdc348894fe7e6221acd
#devlog #programming #rackroot #homelab #python #fastapi #sqlmodel #pytest #refactor #postgresql #backend
I've been on the backend side of Rack Root for so long I just *know* the frontend is out of sync. Oh well, that's a problem for future Aaron to deal with :)
As I'm working on the IPAM table, I now know that I need to remove the gateway field off the subnet class. I don't need to track it there (and I shouldn't) if I already have that data in the IPAM table.
There's a concept in programming called DRY - don't repeat yourself - and I'm applying that here.
I learned something about using Python today, and this might influence how I write other parts of my code. I'm also open to ideas on the best way to go about this.
I'm now most of the way through a refactor where I'm switching to Postgres for my database and also using SQLModel over SQLAlchemy. I have all my classes written/rewritten and was running into an issue on the IPAM side of things when creating a new network. See if you can spot the bug as I write it out here.
Each API call in FastAPI will map to one function, with each class having their functions in their own file, for the sake of modularity/organization. I have a subnet.py file and ipRecords.py for those respective classes.
When a new subnet is created, I make the subnet in the database and then populate all of the IP addresses in the IpAddress table before returning to the FastAPI call. This means that one function call is chained to another one. Specifically subnet.createSubnet calls Iprecords.createIpRecord.
Both functions have a Session object coming in like this. Note that Depends comes from SQLModel and getDb() returns a database session, via yield.
myFunction( input1: int, input2, str, db: Session = Depends(getDb))
This means that when a new subnet is added, and then we call iprecords.createIpRecord, that doesn't get the proper result for getDb and we can't talk to the database.
My fix was to pass the db from one function to another so I can keep using it, and not pull it in via getDb on the inner function call.
Now that I'm through that mess, I can carry on with the rest of my refactor and get back to the main branch.
#pebkac #rackroot #programming #fastapi #python #ipam #homelab #opensource #sqlmodel
After setting up separate Postgres containers, my testing and development is in a really nice spot. I have all the fields working the way I want for DeviceType and Item, all the way to the front end interoperability.
Now I need to add the subnet and IP record definitions back to the project.
I also had to keep the FastAPI validator classes, instead of using the SQLModel definitions directly. I was hoping to simplify that (define classes in less places) but this is working, so I guess I’ll take it. I might research some more how to handle that in a better way, if possible.
#programming #rackroot #homelab #backend #fastapi #postgresql #sqlmodel
Looks like I'm going with Postgres for now. It already has a container I can easily pull down to my local environment to run it. While I'm going through this refactor, I just needed to replace the connection string in database.py and install one additional python package. From there, I was able to get most of my tests passing.
Next time I'm going to be learning some more about the foreign key relationships and how those fit into this whole process.
I want to be able to pull over a property from another table, based on the foreign key, so I will be exploring that. There might be a way to do this with back_populates, which I already have in there. I will need to dig into SQLModel a little more.
I have most of my tests passing (85%!) and I did need to rewrite some of them to account for the ID parameters changing for some reason. I think Postgres and sqlite work a little differently when it comes to inserting records and auto-incrementing an ID. Now that those are fixed, at least I don't have to worry about them.
#programming #rackroot #homelab #backend #fastapi #postgresql #sqlmodel
I think I'm starting to outgrow the sqlite database I've been using. I've been able to get along with some of the foreign key stuff I want to do, but now I'm running into challenges with it accepting DateTime objects. I might need something a tad more powerful.
I'm already halfway through a SQLAlchemy -> SQLModel refactor, I guess what's one more branch on the code?! After all, that's what git is for.
I could probably get away with refactoring the other two classes, but I want all of my testing to be passing before I keep going.
#programming #rackroot #backend #databases #design #fullstackdevelopment #fastapi #sqlite #sqlmodel #pytest
So in about an hour, I refactored out SQLAlchemy and Pydantic to replace it with SQLModel – and some tests are actually passing already! I continue to appreciate pytest and how easy that makes this kind of effort.
Starting with the device type class, I'm going to be working on rewriting my "queries" to fit the SQLModel expectations vs what I was using with SQLAlchemy. I'm using the term queries in quotes there since it looks more like I'm calling methods, and I do see some warnings from SQLModel suggesting what I can replace them with.
From there, I should hopefully be able to simplify my code, classes, data types, tables, and other pieces. That'll most likely be a job for Wednesday.
#rackroot #webdev #backend #database #fastapi #sqlmodel #programming #sqlalchemy #pytest
