I have a pretty awesome computer I bought when I was a baller back in the day. I don’t really use it a lot and prefer to use a laptop, so most of the time the PC is running, but not really doing a lot besides being a repo for all my stuff. I’ve been going through some Terraform tutorials and using vim over ssh which has been fine, but pretty cumbersome. I thought I’d change that.
I knew that there was a way to mount a remote filesystem over ssh but hadn’t done it before. A quick search showed me the sshfs and the syntax is pretty much like scp except you are specifying the mount point for the directory. Easy peasy.
The 2nd thing was syntax highlighting in sublime, my preferred editor. I’ve used Notepad++ in Windows before, and a quick searched showed me something similar,but sublime worked and I decided to stick with it. To add a package, you click Preference -> Package Control & from there select the Install Package option. I searched for Terraform and picked the one that had the most downloads and looked the best (of the 2). Volia!
Finally, after what seems like a really long time, we moved the training server over to the room that is being rented so the people that are in the class have access to the application and reporting tools that we offer. Before we went there, I ssh’d to the machine and opened up virt-manager, shutdown the machines, then shutdown the server. After that we loaded up what we needed and headed over.
You’d think that if we plugged in everything the way it was, it would just work, right? Well, it should have, but didn’t. To make a long story short… libvirtd seemed to load, but we couldn’t virt-manager to connect to libvirtd. So it was basically the worst possible scenario – after testing and everything, it wasn’t just one VM that failed or something similar, it was all of them. So we thought of some options and decided it’d be best to go back to the office where we had more tools and things to work with and test it all there.
I’ll admit, that the reason why virt-manager wasn’t working was something I did. Previously (like a week or so ago), I had installed dnsmasq instead of named to do dns lookups since there was going to be a handful of VM’s running. I’ve used dnsmasq before and it was pretty straightforward. However, with the server running libvirtd it seemed to have a conflict of sorts, so I uninstalled dnsmasq and my counterpart copied over a named configuration over and we used that instead. What I missed it turned out, was libvirtd required dnsmasq. So we got back to work, reinstalled it (left the service disabled), rebooted the machine, and virt-manager was able to connect to it! All was good. There were a couple more things he did to make sure the services started up automatically before we went back to the hotel.
When we got there, we still had a problem. DNS lookups were taking forever. Like after 30 seconds they’d fail. We tried some simple things, like the configuration that was there was a slave configuration so we thought about changing that, and the DHCP service handed out two DNS servers so we removed the extra one and still had the same problem.
I remembered a few weeks ago, when we noticed that DNS was being weird, was I actually got an “OCSP lookup failed” in Firefox. So I checked it out. It turned out, that because the server was off network, and wasn’t able to see if the web servers certificate was valid, it would just not load the page. I found an option to not do OCSP lookups in firefox and that seemed to get rid of the issue at least on my machine and only temporarily. In the training environment though, they had been using Chrome.
We talked about connecting the server to the ethernet port and then feeding it to the wireless routers WAN port so the clients would have internet, but the connection was less than 1 Mbps and they would have internet so they could verify the certificate, but it would be crazy slow and with multiple people doing the exact same thing, it may not have even worked. We did a speed test on their wireless and it was 3 Mbps, so we thought we could get another wireless router, bridge it to the hotel’s wireless, then feed that into our training router since 3 is faster than 1. But that seemed a little too much and too unstable, especially since people literally from all over the country were coming here.
So I had a few things going on in my head while the other Admin was looking at redoing the named configuration (since that’s what we thought it was up to this point. DNS would resolve sorta slow, but once it was cached it was fasts, but the web page for the application still wouldn’t load. At all). So I started looking at open ports to see what was running, used curl to see what it would return. Then tried to get curl to show me the page without checking its certificate – BAM! That worked, I could tell by reading the HTML that came back it was the page I was looking for. Ok so the problem is the certificate (I’d figured this out a few weeks ago as I’d mentioned, but it was at the same time DNS was causing issues and the two got conflated and/or the page was cached on my computer because sometimes it would load).
So I then went to one of the laptops, changed the option in Firefox by going to the “about:config” page and looked for the option that began with ‘security.ssl.’ (I believe) and had OCSP in it, then disabled both of those. It worked. So I relayed this information to him. He thought it was weird that it was working intermittently a few weeks ago and we guessed that it was DNS and OCSP causing different problems that looked the same. So weird. At this time too, he cleaned up the DNS config a bit, I know he removed the extra DNS server since it wasn’t reachable and might have done something else, but I can’t recall exactly.
After I got 1/3rd of the settings in Firefox changed (and let them know to use Firefox instead of Chrome), he found an option on the server to make it so it came from the server instead of having the client have the work around. So he changed it, restarted the service and everything was able to work in the browser that they wanted. Kind of a long day to fix something so simple, but some days are like that I guess. And those kinds of days are the most fun 🙂
I have been going through a few different terraform videos and blogs and got my first ‘real’ terraform example done. It was a walk through from the book “Terraform Up and Running”. I had some bumps along the way – forgot security group, used the wrong AMI, etc. but it’s working now and pretty sweet. Here’s the script I used.
So what it does is create an instance (actually it creates the security group first). The security group is open on port 8080 to and from anywhere. The fun part was the ‘user_data’ since that just opens port 8080 with the contents ‘Hello World’ echo’d into an index.html file.
I mentioned that I ran into an error with the security group not being assigned to the instance. I’ll admit, I didn’t go through the tutorial as carefully as I should have, so I started the instance up and didn’t get anywhere. I figured it was the security group since it wasn’t reachable. I read through the information again, typed ‘terraform apply’ and it came up. Here’s some of the output/screenshots.
When I had to edit the terraform file after adding the security group to get it working
Here’s a snippet of the IP it assigned, showing the output on port 8080
Here’s the security group and rules
And here is other misc information from the deployment
Then at the end I destroyed everything with the ‘terraform destroy’ command and – poof! – in just over a minute it was all gone.
One of the nice things about being a contractor through STG is they have an allotment for $30 a month for continuing education. Some people use it for audible, others for a monthly subscription to a site like pluralsight or safaribooksonline or others. I’ve decided to use mine for purchases from Udemy.
There is quite the little library I’ve been building up, and over the last couple of days, when I’ve had a bit of time, I’ve started going through a course on Terraform, specifically focusing on AWS. It’s been pretty straightforward to go through, and there are a lot of options you can use to configure devices, what order and dependencies, the zones you want them in, type, memory, number of cores, etc. Overall I’m pleased with it. There is another one that’s more in-depth, and after I’m done with this one, I think I’ll go through that one.
I have slowed down studying for the CISSP in the last week-ish. I took some tests last and corrected most of them (and extracted notes from what I missed), but there are 4 more and I’m really just not wanting to sit down and go grade them and get notes from them. It’s like I’m taking the tests again and it’s just a bear to do. I am pretty confident I’m ready, or at least pretty close. I debated for a long time if I wanted to get it before I transitioned over or not, and finally settled on waiting until I transitioned over.
Well, this didn’t take long to happen. I modified the fail2ban settings so after 3 failed attempts the connection would be blocked. I was messing around with xrdp and after testing it on port 3389 from the outside I decided to set up port forwarding so I would connect to my local computer but it would be forwarded to my other one.
This forwards my local port to the remote port on the server that’s listed. I vaguely remembered doing this at one point, but needed a refresher on which part went where.
It all went well and was working – although slightly slower than a connection – for a few times, then inside Remmina, I noticed there was an option to add a ‘pre connection’ command. So I added that ssh command in there and tested it, and it worked! Which was awesome because it saved me from having to type it every time. The one downside was after I typed in my username and password, I actually had to click the ‘Cancel’ button for it to actually connect.
But what ended up happening is that multiple ssh connections stacked up (for some reason, I didn’t bother looking at it) and I think in pressing ‘Cancel’ quickly a few more times, fail2ban must have blocked my IP cause I was no longer able to connect.
Oh well. I guess I don’t ever really need to connect to my home computer from work unless there is something I’m testing. But now I have it available if I need to do it.
I increased the amount of time that people were banned for 10 days and what this inadvertently did was re-ban my work IP. Luckly, I have a lightsail instance up (or I guess I could have spun up a Instance on AWS) and connected to my computer to unban my IP. Normally, I wouldn’t care, but if I didn’t change it, my work IP would be banned for another week. Here’s the command that I used once I logged in, bouncing off my Lightsail instance:
I’ve really liked the idea of having Mattermost running, so originally I set it up on an Amazon Lightsail instance. The least expensive one that would work was $10/month which isn’t a big deal, but at $120/year and with a baby on the way, I didn’t like that. Comcast has been weird ever since they installed their new equipment, but I got a lot of that sorted out now.
Before Comcast installed their new modem, it was Internet -> Modem -> Router -> Home network. Afterwards it was Internet -> Modem (10.0.0.0/24) -> Router (192.168.0.0/24) -> Home network. So there was a double NAT situation and things would work, but wouldn’t and I wasn’t really in the mood to rewire and rethink a bunch of things. But last night I got that sorted out so now my computer is facing the internet and ports are forwarded correctly.
Setting up Mattermost was pretty straightforward. The only things I had to do were to download the certificates and point the mattermost.conf file to them. I set up Mattermost with the appropriate domain name during the install and it was smooth, since I’d messed around with it previously.
Another thing that was easier than I thought was fail2ban. At a previous job they had it set up and everything in the conf file was in weird places. The regular install on Linux Mint was straight forward, easy to read and documented by default (huge improvement). I tested it and got myself locked out after 5 tries and for 10 minutes – I later changed this to 3 tried and 24 hours. The root account isn’t able to log in remotely, so people can bang away at the port and not get anywhere.
Overall, migrating Mattermost over and setting up fail2ban were way too easy.
So this isn’t really something I’m writing to document me fixing a problem, but more of a cool experience I’ve spent the last day-ish on.
Without getting into too much detail, it was decided to archive some old VM’s that we weren’t using and wipe that server for a training environment. It ran Centos 5, had 48 GB RAM, 16 cores, and was perfect because it was older hardware and if it was damaged in transit or whatever (we wanted to make it portable), then it wouldn’t be that big of a deal.
So I archived the VM’s off that machine about a month ago and called it a day.
Then, about 4 weeks later, one of the developers came to me asking why he couldn’t connect to a specific database and I’d let him know that I was directed to archive it. So after talking with the main Admin, we thought we’d try to bring it back, but we needed to migrate everything we had off of Centos 5 anyway (we had at least 1 more machine) so this could be an opportunity do that at the same time.
I did some testing with reloading Centos 5 first on the server we were going to use for training – in the time between deciding to use older hardware and now, we got new hardware and decided it’d be best to use it instead. I got Centos 5 loaded up, tried importing the VM and for whatever reason, it didn’t work. Huh, that was weird. Oh, I did a net-install off a 5.6 disk, better run yum update.
Yum update didn’t work because the repo wasn’t valid (Centos 5 is too old), so I did some digging and had to manually change the Centos-Base.repo to go to 5.11 and then the upgrade worked fine. Downloaded, installed, reboot, boom.
After that, I got virt-manager going and started up a few VM’s and in the setup process, there were two options to go with: Paravirtualized and (Full) Virtualized. Full Virtualized sounded better, so I went with that option, pointed the VM image to the image that I restored and…. it didn’t work. This is the type of message I’d get:
I really don’t like messing with bootloaders, and after a while of digging and reading through searches, etc, I found out that the VM was trying to boot off a boot image that wasn’t there. Then after some more digging, I learned that waaay back in the day when these VM’s were created, they were setup using paravirtualization (the default option when you set up a VM in virt-manager on Centos 5). This image helped me clarify what was going on:
So once I got that straightened out, I reloaded the VM as a paravirtualized VM and it booted up. (I should mention, this was after a lot of testing on the older Centos 5 environment and Centos 7 environment we have, if we could just migrate them to the newer one, it would be great. I did get that to work, but it was on fully virtualized and I thought I just saved us a bunch of time, but to my dismay that eventually wasn’t the case). After it booted up, I got some more errors that prevented the VM from fully booting up (but I was getting close):
So this looked pretty intimidating, but looking at the output, it failed on /dev/xvdb1, which didn’t look like anything in our environment that I’d seen before (more on this later). So I logged in as root in single user mode, I had to edit /etc/fstab – but forgot to remount the root partition first – then rebooted it and it loaded up fine.
Finally! I got the VM up and working, so it should work as expected, right? Not so my friend. An easy thing I had to do first was tell the DHCP server to have this servers (new?) mac address give the restored machine it’s correct IP address. It was getting a new one, it might have been taken out of the dhcpd.conf file, I’m not sure, but it was in the DNS records, so I gave it the same IP.
I logged in, waited for a bit, looked for any process that looked like sql or database or anything, poked around in the normal directories and got nowhere. So I grabbed the main admin, updated him on where I was at told him that I was able to get it to work after I commented that line out of /etc/fstab. Turns out, there was another disk that was needed since this VM ran out of space…. so all the work I did didn’t help the problem that we had, but it was a great learning experience overall.
Oh, I was worried about the loss of a database cause we use them everyday here, and I’m not really great at managing them, but he assured me that he can pull a copy down and recreate what’s needed. Long term, this VM and database won’t be needed soon, but it’s still needed until we formally release a new version and baseline it.