Consistency is important in blogging. It does not matter if you are bad at writing, have nothing to write or nobody reads your blog. What matters is whether you write on a predictable schedule - once a month, once a week, once a year, one per day etc. If you do not have a schedule you are more likely to fall off from sticking to blogging. I had written before, I always struggled to write regularly. But since the start of this year, I have been blogging to a schedule (once a week) and have been successful with that. This post looks into some of the challenges that I faced and things that I have incorporated to stick to a schedule.
Make Writing a Habit
Writing posts was hard and I always procrastinated on it. It often took me more than a week to complete a post. There’s a saying, If it hurts, do it more often - So to make writing easy I had to write more often.
I write only when inspiration strikes. Fortunately it strikes every morning at nine o’clock sharp.
Till a year back I struggled to find topics to blog. I did not find any ‘cool thing’ that was worth blogging. Not much has changed even now on doing ‘cool things’. The only thing that changed was that I decided to write every day - no matter what I do or don’t do.
I have always struggled to find topics to blog, but then I realized all I need is to pay attention to things that I do daily
A year back I asked John Somnez from SimpleProgrammer the same question.
Though I did not get a reply from John, it kept me thinking. Late last year I started a Daily challenge of ‘Generating at least one blog topic a day’ for a month. I did not have to write anything, just find a topic to blog and add it to a OneNote list. I did this for a month and found that I had enough topics to write on. Since the challenge did not have anything to do with writing, I did not restrict my thoughts. This is when I realized that it is not the lack of topics that I don’t blog, but the fear of writing. That’s when I started the ‘Write Everyday’ daily challenge.
This is one of the most important things that I found to be able to write posts regularly - Write Everyday. Yes every day, no matter what happens, I write at least one line towards a post. Since the goal is to Write just one line for a blog post, I hardly miss this goal. Keeping the goal small and simple is a trick that I picked from the book Mini Habits and find it useful. You can follow this for anything that you want to make a habit.
Make it Visible
Seeing something progressing to completion helps me to get it done faster. I used to write elsewhere and then copy it to my blog and publish it. This did not give me a sense of progression. I now have a draft post workflow, integrated with my blogging platform. I can preview this blog along with the draft posts on my laptop. This helps give a sense of progression and also makes the end goal visible.
Making your goal clear and visible helps you achieve it faster
I also use this to keep track of new post ideas. Whenever I find a topic to blog, I add it as a draft. I add bullet points to the post whenever I get more thoughts on that topic. Once there are enough points to write about, I work on that post.
Make Blogging All About Writing
The last thing that I want is to get stuck with mundane repetitive tasks of publishing a post to get in the way of writing. The more regular I became in writing, I started noticing a lot of repetitive tasks in my blogging workflow. When you do any activity repetitively you notice tasks that you do over and over again. I started modifying my blog engine for my needs and to reduce manual effort.
Write in Advance
Having posts ready to be published is helpful as it gives the flexibility to slack a bit when in need. It’s a great feeling to have blog posts ready to be published (right now I have posts ready for a month). I modified my blogging engine to support publishing posts on future dates. This helps me set publishing dates in the future for posts and it gets automatically deployed on the day. Automating the publishing helped make blogging more about writing. I can forget about it as soon as I complete a post by specifying the date I want it to be published and be rest assured that it will get published.
Make Writing Accessible
You never know when you have that free time and want to blog. Maybe it’s while traveling, or waiting in a queue or at your work desk. You might be offline or connected Making it possible to write from irrespective of the kind of device (laptop, mobile, tablet) and connectivity was key for me. I have my draft posts synchronized through Dropbox and available on all my devices. This helps use all the pockets of time that I get for myself.
These are the things that have helped me pick a schedule and follow it. I have been successful in publishing one post per week for the past 10 months and want to keep at it going forward. Now that I am able to publish regularly, I want to improve the speed at which I write posts. Though I reduced the time from over a week to days, I still find this too long and want to reduce it to a day. Modifying the way images are added, optimized and served is also something that I am trying to optimize currently. Do you have a blog? If not start one and stick to a schedule that works for you!
It’s been six months with the Logitech MX Master mouse and I absolutely love it for daily use. I picked this mouse a week after reading Hanselman’s review. It feels just as it sounded in his review - Perfect!.
The mouse fits perfectly into the hand and supports the wrist in a native position. It is suitable for long duration use.
The battery is rechargeable and the mouse comes with a micro-USB charging cable. After a full charge, the mouse lasts me for 2-3 weeks. The mouse can be used while charging, so there is no downtime. The LED lights indicate the charge status. If not the software also warns when the battery is low.
I connect the mouse using the Logitech Unifying Receiver. But the mouse also supports connecting over Bluetooth. This is useful in cases when you want to connect with multiple computers.
The mouse supports Easy Switch technology. You can pair up to 3 devices and switch between them using buttons on the bottom side of the mouse. I have not used this feature as I mostly work on just one laptop.
Darkfield Laser Sensor tracks flawlessly over all surfaces. I use the MX Master across wooden and glass surfaces and find it smooth over both.
The mouse scroll supports smooth scrolling and ‘Ratchet’ mode (click, click, click) scroll. It supports automatic and manual mode shifting. So when reading long documents you can be on smooth scrolling and when navigating code or shorter texts you can use Ratchet mode to feel greater control. It also has buttons to
The mouse has an accompanying software that allows customizations of mouse buttons.
The software allows setting application specific actions for buttons. For e.g. I have the Gesture button to archive emails in Outlook, ‘Navigate to Definition’ in Visual Studio, close tab in Chrome etc. You can configure this to any action on applications of your choice (provided it supports keyboard shortcuts that can be linked to).
The only issue that I faced so long is with the Mouse scroll mode. The ‘Ratchet’ mode no longer works for me. It is always in smooth scrolling mode and the mechanical key to shift modes does not work anymore. Google tells me that this is a common issue with the mouse . This video walks through on how to open up the mouse and fix the mechanical part that clicks in to switch the scroll modes. I have not tried this myself as I don’t feel much a problem without the Ratchet mode.
Other than the Ratchet mode issue, I have not found any other issues with the mouse.
If you are looking for a new mouse and willing to spend a little more than usual, I recommend the Logitech MX Master. What mouse do you use?
* Amazon links are affiliated.
Last month I was in Trivandrum (Kerala, India) for a month’s vacation. This was also my first trip back to India after moving to Sydney. While in India I chose not to have a mobile connection and be offline on the go (at home I had a broadband connection). So while traveling places or moving around I had my mobile in airplane mode. But even then there was time that I resorted to my mobile to spend time. When traveling by bus/train, afternoons in hotels (when everyone rests) etc, I had a lot of ‘me time’.
Below are some of the things that I have my phone geared up for offline mode.
Blogging is one of the things that I enjoy doing most these days. To keep up with my routine, I try to blog every day to keep up with my schedule. I have modified my blogging workflow to give the flexibility to blog from any of my devices. Currently, I am writing this blog while on a train and ‘disconnected’. When back home and connected, all the offline content synchronizes through Dropbox. The content written while offline is now available for publishing from my laptop. Making writing accessible from everywhere helps me to stick with my mini habits.
I always have books, articles downloaded and available for offline reading. For books, I use the Kindle application. Most of the time I have my Kindle for reading, if not I use the mobile application. For blog posts and articles I use Pocket. Whenever I am connected and find interesting posts to read later, I Add to Pocket. Pocket’s mobile application downloads articles and makes them available for offline reading.
I usually do not have many games on my mobile. But in case I feel bored I play the game available on Chrome browser (only in offline mode). When offline a T. Rex dinosaur comes up in the browser with the message ‘Unable to connect to the Internet’. Tapping on the dinosaur starts the game.
Offline time is good to manage your To-Do lists and do a brain dump. I use Todoist to manage my tasks and activities. The Todoist mobile application stores all tasks offline. It allows adding tasks when disconnected and synchronizes it to the server when connected. I also use OneNote for capturing notes which also has an offline mode.
For short local trips, the recently launched Google Trips is useful. Google Now already shows trip summaries and alerts on the home page. Google Trips takes this to the next level and allows to manage all trip data at one place. It helps collate all the travel booking in one place and makes it available in offline mode.
These are the common things that I have my mobile setup for offline mode. I have had this setup while I was in Sydney, as I try to stay disconnected when commuting to work. Staying off the internet helps me get more things done!
Code Formatting is an important aspect of writing code. If followed well it helps keep the code more readable and easy to work with. Here are some of the different aspects of formatting code and my personal preferences. I then explore options to enforce code formatting and ways to introduce it into an existing code base.
Below are some of the popular formatting rules and those that have a high value when enforced in a project.
Tabs vs Spaces
One of the most debated topic in code formatting is whether to use tabs or spaces to intend code. I never knew such a debate existed until my most recent project. It had developers from different parts of the world and with different preferences. I came across the below excerpt from Jeff Atwood, to which I completely agree.
Choose tabs, choose spaces, choose whatever layout conventions make sense to you and your team. It doesn’t actually matter which coding styles you pick. What does matter is that you, and everyone else on your team, sticks with those conventions and uses them consistently.
That said, only a moron would use tabs to format their code.
Settings for these are often available at the IDE level. In Visual Studio this is available under Options, Text Editor, All Languages, Tabs. Be aware of what you choose and make sure you have the same settings across your team members.
Avoid aligning by common separators (=;,) when they occur in adjacent lines. This kind of alignment falls out of order when we rename variables or properties. It happens when you chaange property names.
1 2 3 4 5 6
1 2 3 4 5 6
You should never have to scroll to the right - I caught on with this recommendation from the book Clean Code (a recommended read). It is also recommended that a function should fit on the screen, without needing to scroll up or down. This encourages to keep functions short and specific.
We should strive to keep our lines short. The old Hollerith limit of 80 is a bit arbitrary, and I’m not opposed to lines edging out to 100 or even 120. But beyond that is probably just careless
- Uncle Bob
The Productivity Power Tools extension for Visual Studio allows adding a Column Guide. A Column Guide reminds developers their full line of code or comments may not fit on a single screen.
Aligning Function Parameters
Always try to keep the number of parameters as less as possible. In cases where there are more parameters or longer function names, the team must choose a style. There are different styling formats followed when splitting parameters to a new line.
Allowing parameters to take the natural flow of IDE (Visual Studio) is the simplest approach. This often leads to poor readability and code cluttering.
Breaking parameters into separate lines is important for readability. Use the Column guide to decide when to break function parameters into different lines. There are different approaches followed when splitting parameters into new lines. Keeping the first parameter on the same line as the function and then having all other parameters on new line aligned with the first parameter is another approach. This works well when viewed in the same font and resolution used when writing. When you change font or resolution this kind of formatting falls out of place.
A better variant of the above style is to have the parameters in the new line aligned to the left. This ensures parameters stay in the same place when changing font or resolutions. The one that I prefer is to have all parameters in a new line. This formatting works well with different font sizes and resolutions.
1 2 3 4 5 6 7
Visibility Based Ordering
It is a good practice to maintain a specific order of items within a class. Have all properties declared first, then constructors, public methods, protected methods, private methods etc. This is up to the team to determine the order, but sticking on to it makes the code more readable.
Code Analysis Tools
Checking for styling and formatting issues in a code review requests is a boring task. It’s best to automate style checks at build time (local and server builds). Making build throw errors for styling issues forces developers to fix them. Once developers get used to the rules, writing code without any formatting issues becomes second nature. StyleCop is an open source static code analysis tool from Microsoft that checks C# code for conformance to StyleCop’s recommended coding styles and a subset of Microsoft’s .NET Framework Design Guidelines. It has a Visual Studio plugin and also integrates well with MsBuild.
Cleaning up a Large Code Base
Introducing StyleCop (or any code format enforcement) into a large pre-existing code base is challenging. Turning the tool on would immediately throw hundreds and thousands of errors. Trying to fix them in a stretch might impact the ongoing development process. This often causes us to delay the introduction of such enforcement’s into the project and it continues to be a technical debt.
Taking an incremental approach, fixing one by one as and when a file is changed seems a good idea. Teams can come with the Boy Scout Rule - ‘Leave the file cleaner than you find’. Every time a file is touched for a fix, run StyleCop analysis and fix the errors. Over a period of time, this will make the project clean. The only problem with this approach is developers often tend to ignore/forget running the analysis and fix them.
Trivial things like code formatting is hard to mandate within a team unless it is enforced through tooling
Source Control Hooks
We can plug into various hooks that source controls give to enforce code formatting on developer machines. In git, you can add a custom pre-commit hook to run the StyleCop analysis on all the staged files. StyleCopcli is an open source application that wraps over the StyleCop DLLs and allows running the analysis from the command line. So in the hook, I use this CLI to run StyleCop analysis on all the staged files.
1 2 3 4 5 6 7 8 9 10 11
If there are any StyleCop validation errors the commit is aborted, forcing the developer to fix it. The git hooks work fine when committing from a command line or UI tools like Source Tree. However, Visual Studio git plugin does not run the git hooks and fails to do the check.
Over a period of time, most of the files will get cleaned and remaining can be done all at once with less effort. Once the entire code base passes all StyleCop rules, this can be enforced in the build server. This ensures that no more bad formatted code gets checked into the source control.
Code is read more than written. So it is important to keep it readable and well-formatted. It also makes navigating code bases easier and faster. These are minor things that are often overlooked by developers, but have a high impact on productivity when followed. Do you enforce code formatting rules in your current project? What are the rules that you find important. Sound off in the comments below!
When creating a subscription for a client, the calculated number of months was off by one at times - This was a bug reported from production application that I was currently working on. Though, not a blocker, it was creating enough issues for the end users that it required a hotfix. One of my friends picked this issue up and started working on it. A while later, while I was checking the status of that bug I noticed him playing around with Linqpad. He was testing a method to calculate the number of months between two dates with different values.
We often test our code elsewhere because it’s coupled with other code making it difficult to test at the source itself. The fact that we need to test an isolated part of a larger piece of code is a ‘Code smell’. There possibly is a class or method that can be extracted and unit tested separately.
Having to test code elsewhere other than the source is a Smell. Look for a method or class waiting to be extracted
In this specific case, below is how the code that calculates month difference between two dates looked like. As you can see below, the code is coupled with the newAccount, which in turn is coupled with a few other entities that I have omitted. Added to this, this method existed in an MVC controller, which had other dependencies.
1 2 3 4 5 6 7 8 9 10
This explains why it was easier to copy this code across and test it in Linqpad. It was difficult to construct the whole hierarchy of objects and to test this. So the easiest thing to fix the bug in is to test elsewhere and fit back in its original place.
Extract Method Refactoring
This is one of the scenario where Extract Method Refactoring fits in best. According to the definition
You have a code fragment that can be grouped together. Turn the fragment into a method whose name explains the purpose of the method.
Extract Method Refactoring is also referred in Working Effectively With Legacy Code and xUnit Test Patterns (to refactor test code). It helps separate logic from rest of the object hierarchy and test individually. In this scenario, we can extract the logic to calculate the number of months between two dates into a separate method.
For Test driving the extracted method, all I do initially is to extract the method. As the method purely depends on its passed in parameters and not on any instance variables, I mark it as a static method. This removes the dependency from the MVC controller class parameters and the need to construct them in the tests . The test cases includes the failed ‘off by one’ case ((“25-Aug-2017”, “25-Feb-2018”, 6)). With tests that pass and fail it’s now safe to make changes to the extracted method to fix the failing cases.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
More than the algorithm used to solve the original issue what is more important is in identifying such scenarios and extracting them as a method. Make the least possible change to make it testable and fix step by step.
Whenever there are code fragments that depend only on a subset of properties of your class or function inputs, it could be extracted into a separate method.
1 2 3 4 5 6 7 8
Introduce Value Object
Now that we have fixed the bug and have tests covering the different combinations, let’s see if this method can live elsewhere and make it reusable. The start date and end date on account always go together and is a domain concept that can be extracted out as an ‘Account Term Range’. It can be represented as a DateRange Value Object. We can then introduce a method in the DateRange Value Object to return the number of months in the range. This makes the function reusable and also code more readable. I made the original refactored method as an extension method on DateTime and used it from DateRange Value Object.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
If you are new to TDD or just getting started with tests, introducing tests while fixing bugs is a good place to start. This might also help make code decoupled and readable. Try covering a fix with tests the next time you fix a bug!
Most applications today deals with some form of sensitive information. The most commonly seen are database connection strings, API keys, token etc. The web.config seems the best place to have these values, but it definitely is not. In most cases it gets pushed into the source control systems as well. If it is a private repository then you at least have one level of security on top of it. It still exposes sensitive information to anyone who has access to the repository. It’s worse when the repository is public.
There are different ways you can avoid pushing sensitive data into source control. In this post, I will explore options that I am familiar with.
Use configuration files as template definitions for the configuration data your application requires. Have the actual values stored elsewhere
Azure App Settings
If you are deploying your application as a Web App on Azure, you can store application settings and connection strings in Azure. At runtime, Windows Azure Web Sites automatically retrieves these values for you and makes them available to code running in your website. This removes the need for having sensitive data in the configuration file.
Release Management Tools
Release management tools like Octopus Deploy, Microsoft Release Management, that performs configuration transformation. It supports creating different environments (development, production) and corresponding configurations. On creating a package for an environment, it applies the corresponding environment configurations
Packaging embeds the configuration value into the configuration file. This makes it available to anyone who has access to the host systems.
Azure Key Vault
Azure Key Vault acts as a centralized repository for all sensitive information. Key vault stores cryptographic keys and Secrets and makes them available over a HTTP Api. The objects (keys and secrets) in key vault has unique identifier to retrieve them. Check Azure Key Vault in real world application for more details on how to achieve this. A client application can authenticate with Azure Key Vault using a ClientID/secret or ClientID/certificate. Using certificate to authenticate is the preferred approach. To get Keys/Secret from key vault all you need is the AD Application Id, the client secret or certificate identifier and the key/secret names. The certificate itself can be deployed separately into the application host.
1 2 3 4 5 6 7
If you are using the ‘client secret’ to authenticate then the configuration file will have the Secret. In either cases, you should follow either of the previous approaches to keep the Application Id and authentication information out of configuration. The advantage of using Key Vault is that it is a centralized repository for all your sensitive data, across different applications. You can also restrict access permissions per application.
These are some approaches to keep sensitive information out of source control. What approach do you use? Irrespective of the approach you use, make sure that you don’t check them in!
Since the start of this year, I have been trying to blog to a schedule and publish posts more often. The goal that I have set myself with is to post four posts a month, preferably one each week. I have been sticking to it till now, and I hope it continues. Initially, I did not have this upper limit on the number of posts in a month. In the month of March 2016, I went a bit aggressive and published nine articles. It made me think more about setting an upper limit on the number of posts so that I don’t end up having higher expectations out of myself.
Having published nine posts, also made me realize that I could write faster if required and have posts ready for future. It will help me to stay ahead of the posting schedule and give me some off-time when I need it. But this also presented me with a new problem on how to manage and schedule posts for the future.
The more I automate the mundane tasks of blogging, the more I can concentrate on the writing part
Jekyll Future flag
Octopress is over Jekyll and it provides all the capabilities that Jekyll provides. The future flag in Jekyll indicates whether or not to publish posts or collection documents with a future date. With the flag set to false, Jekyll will not generate posts that have a date in the future. It works perfectly for me as all I need to do is to publish posts into the _posts directory once it’s ready, with a date in the future. I have a draft workflow, which puts posts into a _drafts folder and move them into the _posts folder once ready. I updated the rake script that publishes drafts as posts, to take in a publish date and use that to update the post date.
1 2 3 4 5 6 7 8 9 10 11 12
The rake script appends the publish date to the post file name and also the yaml date information and moves it from the _drafts to _posts folder. It also adds a completedDate set to the current time with the timezone information, just for reference.
Integrating with Travis CI
I have the deployment of my blog automated via Travis CI, which builds and deploys the site when committing to the GitHub repository. For future posts since there might not be a commit on the publish date, I need to trigger the build on those days, to publish the posts scheduled. The Azure Scheduler enables scheduling requests and also provides out of the box support to invoke web service endpoints over HTTP/HTTPS. Travis CI exposes an API to interact with the build system and is the same API that the official Web interface uses. The API supports triggering builds by making a POST request with an API token and the build details. The API has an existing bug that requires the slash separating username and repository name in the trigger URL be encoded(%2F). Azure, however, does not like this and treats it as an invalid URL with the bellow error.
The only way now is to have to custom write this code and have it scheduled. I chose the one with the least work involved - Azure Automation. Azure Automation allows to create Run books and automatically trigger it on a schedule. The Azure Automation has a pricing plan with 500 minutes free Job run time in a month, which meets my requirements. I created a PowerShell script and added in the token (TravisToken) and the build URL (TravisBuildUrl) as parameters to the script.
1 2 3 4 5 6 7 8 9 10 11 12
The script runs on a schedule every day and triggers the Travis CI build. It deploys the latest generated site to Azure WebApp that hosts the site. Any posts scheduled for the current date gets picked up by Jekyll and included in the site generation.
Post to Social Media
With the posts getting deployed automatically, I want to update all my social networks. I already use Buffer to post updates to all social networks. Buffer is like ‘Write Once, Post Everywhere’ kind of service. It clubs all your social media profiles into one place and allows you to post to all of them by just writing it once.
IFTTT(‘If This Then That’) is a service that helps connect different apps and devices together based on a trigger. As the name says, you can trigger an action based on a trigger. IFTTT has many Channels that can act as a source of the trigger. In my case, the trigger is a post getting published and I can hook into that event using the Feed Channel. The feed channel has an option to trigger when a new item is available on the feed. I use this to trigger an update to Buffer. Buffer is available as a channel on IFTTT but allows only update to one of the connected accounts in Buffer, which requires me to setup a recipe per social media account. I chose to use update via email feature in Buffer. It allows me to have just one recipe in IFTTT to update to all of my connected profiles in Buffer.
With the Automated publishing of posts and ability to schedule them, I can concentrate more on just the writing part. I no longer have to push out posts manually. I had never thought that I would be scheduling posts in the future. But now that it is happening it’s a great feeling when there are posts for a few weeks ahead all ready to go.
My Morning Routine was the first posts to be deployed using the schedule.
Local Build Environment with Docker
I am on an older repository fork of Octopress and have not updated to latest version. So it has hard dependencies with specific versions of gem packages that it needs and also on the Ruby and Jekyll version. So every time I change laptop it’s difficult to set up the blog environment. In the past, I manually installed the dependencies whenever I got a new laptop. As changing laptop does not happen frequently, I had been delaying creating any script for this. But now since I had to setup the Travis build environment, I thought of also having a local build environment to test before pushing it up to Travis. Travis provides a Docker image that matches exactly their build environment.
Setting up Docker is just a few steps:
Once in the container, you can run the same build scripts that you manually run yo deploy and check. I had a few issues with the gem packages and fixed it by specifying hard package dependency. To launch the site hosted in Docker from host system I expose incoming ports through the host container. Once I have the local server running in the docker container (in port 4000) I can access it via localhost:4000 from my host computer.
Post Dates and TimeZones
When building from the container, I noticed that the dates of posts were off by 1. For posts that were on month start (like Aug 1), it started coming up in July, on the archive page. After a bit of investigation, I realized that Jekyll parses the date time from the post and converts them into local system time. The container was running in UTC and when generating the site it converted post DateTime to UTC. All the posts that I had written after coming to Sydney had an offset of +1000 (or +1100) and most were published early in the morning. So it converted those posts to the previous date. Since I am not that worried about the time zone of the post, I decided to remove it. I removed timezone information getting set for new posts in my Rake scripts. For the existing posts, I removed all the timezone information from the datetime YAML header in the posts. I set the config.yml to built in UTC irrespective of the system timezone that it is getting build.
Setting up TravisCI
Setting up automated build on Travis CI is smooth and easy process TDK. I just added a travis.yml with the ‘rake generate. TDK The post build script does the following
- Clones the current statically generated code from my blog branch.
- Perform a rake deploy that updates the cloned code above with the latest site.I updated the existing rake deploy to use GitHub token in push URL. As I did not want the token to be logged on to the Travis console I redirect the output using &> /dev/null.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Every time I make a commit to the GitHub master branch, the automated build triggers and deploys the latest generated site.
- Write posts on my phone or laptop. (Using Dropbox to sync posts across devices)
- Publish and Push to Github from laptop
- Travis builds triggered by Github webhook
- Travis pushes back generated site into Github (blog branch).
- Azure Web App triggers automated deployment from Github.
With the automated deployment, I have one less thing to take care of when writing posts. The whole process might feel a bit complicated, but it is not. It is just that I have been tweaking few things to ease blogging. And since I am a programmer, I like hacking things. If you are new to blogging you do not need them and don’t get overwhelmed (if at all you are). All you need to make sure is to have a blog and you own the URL.
Yesterday I was late to leave office as I had to data fix some of the systems that we are currently building. We had just migrated few hundred clients onto the new platform. Invoices generated for the clients had wrong invoice amounts due to some mismatching data used when migrating. We had the expected invoice from the old system which made finding the problem easy. We ran a few scripts to correct the data in different systems and fixed the issue.
It All Starts with a Few
I have seen it repeatedly happen that this kind of data fixes starts with a few in the beginning. Within a short span of time the affected data size grows drastically and manual updates might not be a good solution.
If you get a second thought of whether to script the fix or not, then you should script it.
Yesterday it started with data fix for 30 clients and the fix was relatively small. It could either be through UI or API. Fix through the UI took around 45 seconds each, and there were two of us. So it was just a matter of 12-15 minutes to fix it. While fixing, one of us found an extra scenario where the same fix needs to be applied. Re-running the query to find such clients bombarded the number to 379. At this moment, I stood up and said I am going to script this. There is no way I am doing this manually. Manually fixing this would take five man hours, but will finish in two and half hours, as there were two of us. Even writing the script is going to take around an hour but that’s just one man hour.
There is happiness you get when you script the fix and not manually plow through the UI fixing each of them
The script was in C#, written as a test case, invoked from a test runner (which I don’t feel great about now) updating the systems with the data fix. It did its job and fixed all the cases it was supposed to. But I was not happy with the approach that I had chosen to make the fix. Correcting production data through a unit test script does not sound a robust solution. The reason to choose tests was that the test project had all the code required to access the other systems. It was just about changing the configuration values to point to the production system. It was the shortest path to having at least one client updated and verified.
Having it as a test script restricted me from scaling the update process (though I could have done some fancy things to run tests in parallel). It also forced me to hard-code the input data.Logging was harder and I used Debug.WriteLine to the VS output window. All those were the aftermath of choosing the wrong execution method - running it as a test script!
In retrospective, here are a few things that I should have done different and should be doing if ever I am in a similar situation again.
Create Stand-alone Executable
Having a stand-alone executable running the script provides the capability to scale the number of processes as I wanted. Input can be passed as a file or as an argument to the application allowing to break the large data set into smaller subsets.
Log Error and Success
It’s very much possible that the ‘fix-to-fix errors’ can go wrong or throw exceptions. So handle for errors and log appropriate message to take any corrective actions. It’s better to log to a file or other durable storage as that is more foolproof. Logging to the output window (Debug.Writeline/Console.Writeline) is not recommended, as there is a risk of accidentally losing it (with another test run or closing VS).
Logging successes are equally important to keep track of fixed records. It helps in cases where the process terminates suddenly while processing a set of data. It gives a track of all data sets that were successfully processed and exclude from following runs.
It is very likely that the script has bugs and does not handle all possible cases. So as with any code, testing the data fix script is also mandatory. Preferably, test in a development/test environment, if not try for a small subset of input in the production. In my case, I was able to test in the development environment and then in production. But still, I ran a small subset in production first and ended up finding an issue that I could not find in development.
Parallelize if Possible
In cases where the data fixes are independent of each other (which likely is when dealing with large data fixes), each of the updates can be in parallel. Also using nonblocking calls when updating across the network helps speed up the process, by reducing the idle time and improves the overall processing time.
Parameterizing of input to the script (console) application helps when you want to scale the application. In my case updating each of the clients took around 8-10 seconds as it involved calling multiple geographically distributed systems. (Updating a system in the US from Australia does take a while!). Having a parameterized application enables to have multiple applications running with different input sets updating the data and speeds up the overall processing time.
It’s hard to come up with a solid plan for critical data fixes. It might not be possible to follow all of the points above. Also, there might be a lot other things to be done other than these. These are just a few things for reference so that I can stop, take a look and move on when a similar need arises. Hope this helps someone else too! Drop in a comment if you have any tips for the ‘eleventh hour’ fix!
Humans are creatures of habit and things work well if made as a routine. It’s what you build as your daily plan that defines what you end up achieving in the day and in turn with life.
If you are a late night person, you can read ‘morning’ as ‘late night’ - the focus here is routine!
A morning routine is nothing but a sequence of actions regularly followed. I have tried switching my routine too, to late in the night a few times but found that mornings work better for me. But this could be different for you, so stick to the time of day that works for you. Before going to how my morning routine looks like (which I just started a week back), I will explain how I made the plan for the routine.
At any point in time, there are a lot of things in my mind and things that I kept committing to myself and others. It is not possible to keep up with everything that I wish to do. So the very first thing to do is to dump everything out onto to paper and then decide what needs attention. The Incompletion Trigger List assists to get everything out of your mind onto paper. It’s a good idea to block some time of yours to perform this exercise and give it all the attention it needs. At times it helps to Slow Down to Go Fast.
Most Important Task (MIT)
If you are following along, hope the brain dump helped to flush out all that was there in your mind. This exercise needs to be occasionally done (maybe every 2-3 months) to stay clear and stay on top of things. I was sure that I could not do everything on that list after the brain dump. Now comes the hard part of choosing what matters to you and choosing those that aligns well with your goals. For the morning routine, I stuck to items from the brain dump that fall under ‘Personal Projects’ category (as highlighted in the below image).
These are the items that matter to me and aligns to the short-long term goals that I have. Below is a part of my list.
1 2 3 4 5 6 7
The key is not to prioritize what’s on your schedule, but to schedule your priorities.
– Stephen Covey
Progressing towards all of these items on the list at the same time is not possible, as time available each day for achieving them is limited. I usually get around 2-3 hours a day of ‘me time’, provided I wake up at 4 in the morning (more on this shortly). The number of hours you have might differ, and you could choose those many items as you think you can fit in. But 3 is a good number to choose, as that helps to mix in a few different goals and gives the flexibility to shuffle around with them on a day. For me, it also means I roughly get around 40-60 minutes daily, for each item.
Currently, the ones that I have in my morning routine are:
- Learn Functional Programming
- Open Source Contribution
MITs to Mini Habits
Having high-level short-long term goals is good, but does not provide anything actionable on a daily basis. It feels overwhelming to approach them because it does not give any sense of direction. So it’s important that I have small actionable items that I can work on and progress towards achieving the goal.
Break your goal into the smallest possible task that you can think of so that you don’t feel to skip it
For me, the mini habits look like this
- Write at least one sentence for the blog
- Read at least one line about Functional Programming
- Read at least one line of code of an Open Source Project
The idea behind keeping it so small is just to start. It’s very rare that I have stopped writing after writing a sentence or stopped reading after a line. The trouble is only with getting started - once done you can easily carry on for at least 20-30 minutes. Even if I make 2 out of the 3 of the above tasks, I take it as a success, which gives me some flexibility each day.
Waking up Tricks
There are days when Resistance beats me to it, and I don’t get up to the routine. But I feel low on those days for not able to progress on my goals. So I try hard to have less of such days.
Alarm Phone Inside Pillow: I use Timely on a spare phone to set alarms. Till a while back I used to keep the phone at the bedside while sleeping. But I noticed that I often ended up snoozing the alarm, at times even without full consciousness. So to make I wake up to the alarm, I now keep the phone buried inside my pillow with just the vibration. The vibration forces me to wake up and also removes the need for any alarm sound - my kid and wife does not get disturbed.
Wear a Sweater: During winter, at times the cold beat me to it. It’s hard to leave all the blankets and wake up to the cold. I started sleeping with the sweater, and I don’t feel that cold when I wake up.
Rationalize Against Resistance: However hard I try not to rationalize on getting up when the alarm sounds off, looking at the snooze button I end up rationalizing. Often I have found that when I try to use the tasks that I can achieve if I wake up, to motivate myself, I end up justifying that it can wait for tomorrow. Because there is no hard deadlines or accountability to anyone - it’s just me!. Now I try just the opposite - Think about the resistance that is trying to force me to the bed and reassure myself that I should not fall to it. The ‘me’ waking up after sleeping in is not going to like it then. So wake up!
- Wake at 4 am
- Drink Water
- Review the tasks for the day (Todoist)
- Mini Habits (2 or 3)
- Wake up wife at 5:45 am
- Continue Mini Habits (2 or 3)
- Wake up Gautham at 7 am
Having a morning routine has helped me focus more on things that matter and not wander from one task to another. It has also helped set a sense of direction to what I do every day and spent less time in thinking what to do. I find my days starting in gradually and not rushing into it, setting up the pace for the day. Hope this helps you too!