Developer Blog Archive

Hello Habitican and faithful blog reader! If you’re following our WordPress site, you may have received alerts that we’ve added some entries from our now-defunct developer blog. We’re collecting these posts on this blog so that they are still accessible to anyone who may be interested.

Today we’ve also posted a new developer entry, regarding the partial outage that affected Parties and Guilds a few weeks ago. You can read this new entry here.

If you’re interested in reading the past developer blog entries we added, you can find them collected here.

Thanks as always for your interest, and for being a part of Habitica!

Overview of Changes in Version 3 of the API

Hello Habiticans! We’re moving our posts from our defunct dev blog, which was hosted at devs.habitica.com, to this blog. Note that much of the information in these posts is outdated, and we preserve them here for historical purposes as well as in case they contain any useful information for posterity. Some posts have been omitted as they will not be relevant for those purposes. Please reach out to us if you’d like to receive a list or document containing a complete record of the devs.habitica.com content.

Important info! Version 2 of the Habitica API is now deprecated, so all open-source developers should switch over to version 3 of the API as soon as possible. To make this transition easier for our developers, we are still supporting API v2 for a few weeks, but we will be shutting it off in the near future. The sooner developers can migrate to the new API, the better! We will announce the date of the version 2 cutoff shortly, so stay tuned.

Server Responses

Successes

Just about every server responses will return a JSON object (exceptions for routes that return other data types, such as the csv export route). You can expect each response to include a success property with a Boolean value and a data property that is an Object or an Array which includes any relevant information back from the request. For instance, the GET /api/v3/tasks/userroute would return data that looks like this:

{
  "success": true,
  "data": [
    {
      id: 'the-task-id',
      type: 'habit',
      text: 'task text',
      notes: 'task notes',
      // more task properties
    },
    // More task data
  ]
}

Some routes also include a message in the user’s language. If your integration has a GUI, this can be helpful for presenting a confirmation message to the user that their action was a success.

{
  "success": true,
  "data": {
    // Some data
  },
  "message": "A message in the user's language"
}

Errors

If the request returns an error, the success property will be false, and there will be an error property with the name of the error and a message property with some info about the failed request in the user’s language.

{
  "success": false,
  "error": "NotFound",
  "message": "The task could not be found."
}

In the case of validation errors, there may also be an errors property which is an Array of error messages.

Changes to API v2

The backend code for API v2 had to be changed to cope with database structure changes. However, there should be no behavior changes, except that certain new fields will be in the data. If you notice any other changes, please report a bug.

The remainder of this post describes changes in API v3, except where otherwise indicated.

Changes to the User and Tasks

In version 2, tasks were stored directly on the user object. In version 3 of the API, tasks are their own collection. The user object now has a tasksOrderproperty which contains a list of IDs for each type of task and you can use this to determine what order the tasks should appear in.

Completed To-Dos do not appear in the tasksOrder lists (see “Completed To-Dos below”).

Changes to Tasks

Task ID Uniqueness

In version 2, each user’s tasks had unique IDs, but the same task ID could be used in tasks for other players. In version 3, all task IDs are unique across all players. See also “Task IDs have Changed” and “Changes to Challenge Tasks” below.

Task IDs have Changed

Because of the requirement for unique task IDs across all users, some of your task IDs may have changed. If your task ID was a valid UUID and was unique across all players, the ID remained the same. Otherwise, a newly generated unique ID replaced the original ID of the task. Update: This was our original plan, but we ended up just creating new uuid’s for each task that was created.

For tasks with invalid IDs that existed before API v3 went live, the invalid old ID is stored in the _legacyId field, and this field is returned by both version 2 and version 3. This field will be removed when API v2 is decommissioned.

API v2 also returns the legacy ID in the id field so you do not need to immediately update any scripts or configuration files that have hard-coded IDs. However you will need to do that when API v2 is decommissioned.

The _legacyId field does not exist for tasks created after API v3 went live.

Custom IDs

In version 2, you could set a custom ID for the task (such as productivity). Because tasks are now their own collection, they must have a unique ID, so we’ve decided to drop this functionality.

When using version 2 of the API, you can still score tasks with a custom ID. When you fetch tasks with version 2, the custom ID will appear under the idproperty, but the database ID will be under the _id property. When using version 3, you can still specify a unique ID for your task when creating it, but it must be a valid UUID. If the ID already exists, the creation of the task will fail.

dateCreated / createdAt

We’ve changed the name of dateCreated to createdAt. (dateCompleted has not changed.)

updatedAt

Each task contains an updatedAt date field, initially equal to createdAt. It is updated when the task is edited or scored/completed/purchased, and when cron changes the task’s value (cron does not affect that field for Habits that have both + and -).

Tags

Version 2 contained tag IDs as an Object. Version three contains tag IDs as an Array.

Completed To-Dos

In version 2, completed To-Dos were included when all tasks or all To-Dos were returned. In version 3, they are not returned when all tasks are requested with GET /api/v3/tasks/user or when all To-Dos are requested with GET /api/v3/tasks/user?type=todo.

Completed To-Dos can be returned using only GET /api/v3/tasks/user?type=completedTodos. They are returned in order of completion date, with the oldest first. They can not be sorted with /api/v3/tasks/:taskId/move/to/:positionand do not appear in the tasksOrder lists.

Currently, only the 30 most recently completed To-Dos are returned. Soon there will be an option to return all completed To-Dos at once or an option to return all completed To-Dos in batches of 30.

Changes to Challenge Tasks

Challenge Task IDs

In version 2, when you joined a challenge, each task you were given had the same task ID as the copy in the parent challenge’s document (i.e., all participants had a task with an identical ID). In version 3, task IDs are unique across all users, so when you join a challenge, each task you are given has a new ID.

The ID of the task from the parent challenge is stored in the challenge object as taskId, along with the id of the challenge itself.

{
  "success": true,
  "data": [
    {
      id: 'unique-task-id',
      challenge: {
        'taskId': 'id-of-task-in-parent-challenge',
        'id': 'id-of-challenge'
      },
      // more task properties
    },
    // More task data
  ]
}

Challenges

In version 2, challenges contained a members field with a list of user IDs. In version 3, they do not.

The user object contains a challenges field with an array of challenge IDs.

See also “Changes to Challenge Tasks” above.

Groups (Guilds and Parties)

In version 2, groups contained a members field with a list of user IDs. In version 3, they do not.

When a user is in a party, the user object contains a party._id field with the group ID of the user’s party.

The user object contains a guilds field with an array of guild IDs.

Miscellaneous Changes

The transformation buff spookDust has been renamed to spookySparkles. All user accounts have been updated appropriately. This applies to API version 2 and 3. If you use spookDust as the skill key in the v2 route, it will still work.

The healer skill with the key heallAll has been corrected to healAll. If you use heallAll as the skill key in the v2 route, it will still work.

Miscellaneous Additions

Several new routes exist. See the API docs (available after the scheduled maintenance is over) for details.

Closing Remarks

We know these changes are significant, and that it will take some time to upgrade your integrations to be compatible, but we are very excited about the future features and integrations that we can develop using this improved API. If you have any questions, please feel free to reach out to us via the Aspiring Coders Guild and we will be happy to help.

Thank you so much for your patience, for your enthusiasm, and for being part of our open-source community. You all are the lifeblood of Habitica, and we’re lucky to have you battling at our sides.

Post-Mortem: Android Outage

Hello Habiticans! We’re moving our posts from our defunct dev blog, which was hosted at devs.habitica.com, to this blog. Note that much of the information in these posts is outdated, and we preserve them here for historical purposes as well as in case they contain any useful information for posterity. Some posts have been omitted as they will not be relevant for those purposes. Please reach out to us if you’d like to receive a list or document containing a complete record of the devs.habitica.com content.

tl;dnrA change to the API related to how collection quests are handled caused the Habitica Android App to cease functioning. The issue is now resolved.

Stats

  • Outage Time: June 6th, 2016 @ 12:56pm UTC – June 7th, 2016 @ 2:05am UTC
  • Percentage of Active Android Users Affected: 57%

Why did this happen?

In an effort to fix a pernicious quest collection bug, we changed the API to determine what items were found in the collection quest at cron instead of when scoring a task. This allowed us to use the party’s data to determine what quest the user was on, rather than the user’s, which could get out of sync.

Part of this change involved changing the user object’s user.party.quest.progress.collect property from an Object to a Number. Since these were properties that were only used by the server at cron, we didn’t think much about changing them.

Unfortunately, even though the Android app was not using that property, it was still mapping that attribute and specifying that it should be an Object, not a Number. When it got data that was not how it expected, it showed a blank profile, which made the app unusable.

We have Fabric installed on the Android app, and we thought it would have reported to us when this many failures were happening, but for some reason it did not. We are still investigating why that happened. As a result, we did not know about the issue until around 9:00pm UTC on Sunday.

How we fixed it

Alys was the first one to notice, and she immediately began working on a fix. In short, we set user.party.quest.progress.collect back to an Object and created a new property to track collection items, user.party.quest.progress.collectedItems, which is a Number.

This lets us deprecate user.party.quest.progress.collect without breaking backwards compatibility. The progress.collect property will be fully removed in the next major version of the API.

After extensive testing, we deployed the change and ran a migration file to copy over everyone’s collection progress to the new property and set the old property to an empty object.

What we are doing to prevent this in future

  • We’re investigating why we were not notified via email by Fabric
  • We’ve set up better alerting for errors like these, so the whole team can see when there are problems and help troubleshoot together
  • We’re going to work on setting up automated integration tests that run mobile apps against the latest version of the server code to catch issues like this before they happen

An Apology

If you were affected by this outage, we’re very sorry. We know that having Habitica unavailable is frustrating and disruptive. We want to thank you for your continued support. We’re working hard to prevent this from happening again.

API v2 -> v3 Migration Guide

Hello Habiticans! We’re moving our posts from our defunct dev blog, which was hosted at devs.habitica.com, to this blog. Note that much of the information in these posts is outdated, and we preserve them here for historical purposes as well as in case they contain any useful information for posterity. Some posts have been omitted as they will not be relevant for those purposes. Please reach out to us if you’d like to receive a list or document containing a complete record of the devs.habitica.com content.

Although version 3 of the API is an entire rewrite, for most of the API routes, there is very little you have to change. For an in-depth overview, read this.

For a real life example, I upgraded my Node API wrapper to use v3 of the API.

Step 1

The most obvious change is make your base URL https://habitica.com/api/v3/instead of https://habitica.com/api/v2/.

// Old
var BASE_URL = 'https://habitica.com/api/v2/'  
// New
var BASE_URL = 'https://habitica.com/api/v3/'  

Step 2

Many of the route urls stayed exactly the same, but a few changed. Most notably, almost all the /user/tasks routes changed to /tasks.

Note: One exception to the above statement is if you want to get all of your user’s tasks, you can use the GET /tasks/user route.

See the API v2 docs and compare them to the API v3 docs for other changes.

// Old
var userTaskUrl = BASE_URL + 'user/tasks';

// New
var userTaskUrl = BASE_URL + 'tasks/user'  

Step 3

The result of a successful request will contain a data property with the information returned from the request. In most cases, this data will be exactly the same as the data returned from version 2 of the API. So, in most cases, all you need to do is grab the data from the data property.

// Old
request.get(userTaskUrl, (err, response) => {  
  response; // an array of tasks
});

// New
request.get(userTaskUrl, (err, response) => {  
  response.data; // an array of tasks
});

Step 4

The result of an unsuccessful request will contain an error property indicating the type of error and a message property translated into your user’s specified language that you can display to the user.

// Old
request.get(userTaskUrl, (err, response) => {  
  if (err) {
    // unpredictable format, different for many routes
    console.error(err); 
  }
});

// New
request.get(userTaskUrl, (err, response) => {  
  if (err) {
    // A translated message about the error 
    // that you can display to your user
    console.error(err.message); 
  }
});

Step 5

Share your integration with us in the Aspiring Comrades Guild!

API v2 Deprecation Notice

Hello Habiticans! We’re moving our posts from our defunct dev blog, which was hosted at devs.habitica.com, to this blog. Note that much of the information in these posts is outdated, and we preserve them here for historical purposes as well as in case they contain any useful information for posterity. Some posts have been omitted as they will not be relevant for those purposes. Please reach out to us if you’d like to receive a list or document containing a complete record of the devs.habitica.com content.

On May 21th version 3 of Habitica’s API was released with many improvements over version 2, in order to enable to the development of new features we’re going to deactivate API v2 on July 15th.

After that date API v2 endpoints won’t work anymore so if you have an integration that still hasn’t been updated to work with version 3, hurry up!

Docs for API v3 are available at https://habitica.com/apidoc. We also have a migration guide.

If you have any questions or need help, don’t hesitate to contact us on Github or in the Aspiring Comrades guild.

Important notice about the migration from API v2 to v3

Hello Habiticans! We’re moving our posts from our defunct dev blog, which was hosted at devs.habitica.com, to this blog. Note that much of the information in these posts is outdated, and we preserve them here for historical purposes as well as in case they contain any useful information for posterity. Some posts have been omitted as they will not be relevant for those purposes. Please reach out to us if you’d like to receive a list or document containing a complete record of the devs.habitica.com content.

The Overview of Changes in Version 3 of the API post, in the section about tasks IDs, stated this:

Because of the requirement for unique task IDs across all users, some of your task IDs may have changed. If your task ID was a valid UUID and was unique across all players, the ID remained the same. Otherwise, a newly generated unique ID replaced the original ID of the task.

We’ve just realized that this isn’t correct and that instead all the tasks IDs used in API v3 have been randomly generated from scratch. This means that apps or extensions relying on tasks IDs remaining the same between API v2 and v3 will have to prepare for IDs to change between API v2 and v3.

So in most cases, the easiest way to update the ids is to fetch all the tasks using the GET https://habitica.com/api/v3/tasks/user route, loop over the array of tasks, check if a task has a _legacyId, and if it does, replace the stored id in your integration with the id field from the task.

Here’s an example for how to do this in Node:

var request = require('superagent');  
var YOUR_USER_ID = 'Your User Id Goes Here';  
var YOUR_API_TOKEN = 'Your API Token Goes Here';

request.get('https://habitica.com/api/v3/tasks/user')  
  .set('x-api-user', YOUR_USER_ID)
  .set('x-api-key', YOUR_API_TOKEN)
  .end(function (error, result) {
    if (error) {
      console.error('An error occurred with the request');
      console.error(error);
      return;
    }

    var tasks = result.body.data;

    var tasksToUpdate = tasks.filter(function (task) {
      return task._legacyId;
    });

    tasksToUpdate.map(replaceId);
  });

function replaceId (task) {  
  var oldId = task._legacyId;
  var newId = task.id;

  // check if old id exists in stored ids
  // if it does, replace with new id
}

Due to this discovery we’ve decided to keep API v2 active until July 30th instead of the planned July 15th to ease the transition from v2 to v3.

We’re sorry about any problem this may have caused.

Outage July 26, 2016 Post-Mortem

Hello Habiticans! We’re moving our posts from our defunct dev blog, which was hosted at devs.habitica.com, to this blog. Note that much of the information in these posts is outdated, and we preserve them here for historical purposes as well as in case they contain any useful information for posterity. Some posts have been omitted as they will not be relevant for those purposes. Please reach out to us if you’d like to receive a list or document containing a complete record of the devs.habitica.com content.

Today at around 1:50pm Pacific time we had a brief outage where the site would display as a white screen and the API was unreachable. The cause of this was some new code added to the site that worked fine in development, but caused errors in production. Thankfully, our server monitoring caught the issue immediately and we were able to rollback to the previous version of the site within minutes.

Some Background

As part of the work we did to write the 3rd version of the API, we adopted Babelas a build tool, which allows us to write using features that have been confirmed for use in the Javascript language, but are not yet present in it.

In development mode, we transpile the code on the fly. This is great for working locally, because we don’t have to wait for a transpilation step to occur, but it’s not great for running a production application, so we transpile all the code at once before the application starts.

However, there is one particularly tricky part of our code base, the files that are shared between the server and the client. In order for this dual set up to work, the server must require these common files from a single entry point so it’s referencing the right versions of the files.

So What Happened?

We put out some code in the server that referenced a particular file in the common/script directory. It worked fine in development, because we were transpiling all those files on the fly, but in production, the Node server choked on some syntax it didn’t recognize. The fix was as easy as changing the path.

Only a few of us were aware of this quirk of our setup, and those of us who knew about it did not catch the mistake until it was too late.

Will this Happen Again?

I’ve added a sanity check that no code in the server will reference any of the files in the common/script directory directly. So if the mistake does happen again, our automated tests will catch it before it is deployed.

Thanks for reading and being awesome Habiticans!

Require-Again (and again and again and again)

Hello Habiticans! We’re moving our posts from our defunct dev blog, which was hosted at devs.habitica.com, to this blog. Note that much of the information in these posts is outdated, and we preserve them here for historical purposes as well as in case they contain any useful information for posterity. Some posts have been omitted as they will not be relevant for those purposes. Please reach out to us if you’d like to receive a list or document containing a complete record of the devs.habitica.com content.

There are some parts of the Habitica that have different behaviors when running the app in production and running the app locally in development. For instance, development mode:

  • has debug routes to make QA testing easier
  • does not send transactional emails (Log-in reminders, flag message, etc)
  • does not send analytics information

However, testing these behaviors can be difficult because Node’s module system caches the files whenever you require them. So if I have a file like this:

// my-module.js
import nconf from 'nconf';

const IS_PROD = nconf.get('IS_PROD');

if (IS_PROD) {  
  module.exports = realFunction; // does something
} else {
  module.exports = noopFunction; // does nothing
}

When testing, I can’t check production versus development behavior, because Node will cache the module in the state it was when it was first required.

In fact, if another test file requires a lib that uses this module, we’ll get the file in the state it was during those tests.

For this reason, I wrote require-again. A module that returns a freshly required version of the module.

Here’s an example of how you could use it:

import sinon from 'sinon';  
import nconf from 'nconf';

describe('myModule', () => {  
  context('production', () => {
    before(() => {
      sinon.stub(nconf, 'get').withArgs('IS_PROD').returns(true);
    });

    after(() => {
      nconf.get.restore();
    });

    it('does something', () => {
      let myModule = requireAgain('./path/to/my-module');

      let result = myModule();

      // assert on result
    });
  });

  context('development', () => {
    before(() => {
      sinon.stub(nconf, 'get').withArgs('IS_PROD').returns(false);
    });

    after(() => {
      nconf.get.restore();
    });

    it('noops', () => {
      let myModule = requireAgain('./path/to/my-module');

      // assert that it noops
    });
  });
});

How do you deal with this challenge? Let us know in the JavaScript guild.

Zapier Integration

Hello Habiticans! We’re moving our posts from our defunct dev blog, which was hosted at devs.habitica.com, to this blog. Note that much of the information in these posts is outdated, and we preserve them here for historical purposes as well as in case they contain any useful information for posterity. Some posts have been omitted as they will not be relevant for those purposes. Please reach out to us if you’d like to receive a list or document containing a complete record of the devs.habitica.com content.

The Zapier connection with Habitica is now in beta!

If you’re not familiar with Zapier, it’s a platform that allows you to connect apps together. So if you need to link together Habitica with Google Calendar, Todoist, Slack or many other services, this is your lucky day!

Here’s a step by step guide for getting started with Zapier. For this example, we’ll walk through how to create a new todo every time someone posts to your party’s chat. It’s a bit of a contrived example, but it’ll show you how to set up Zapier and allow you to swap out the trigger and action with whatever apps you want to use.

Step 1 – Sign up for Zapier

If you already have an account, you should just login instead and go to step 2. 🙂

Otherwise, head over to zapier.com and fill in the form on the home page to get started.

Step 2 – Join the Habitica Beta

Most apps on Zapier are public, so you can search for them in the app selector. Habitica’s integration is still in beta, which means you need to follow this invite link to get access to it. Zapier requires at least 10 beta users before we can make the app public. Once it’s public, new users will be able to find it from the app selector.

Follow the button that says Accept Invite and get ready to make a zap.

Step 3 – Setup the Trigger for your Zap

To get started, press the “Create this Zap” button. This will bring you to a page where you’ll be able to choose a trigger. The two options available now are Task Activity (trigged when a task is created, updated, deleted or scored) or New Group Chat (triggered when a new message appears in a group chat).

We’ll choose New Group Chat.

The next page will have a button that asks you to Connect Your Account. Press the button and a new window will appear that prompts you for your user id and API token. You can find these credentials on the settings page.

Next, you’ll need to choose a group. For this example, we’ll choose our party.

The final page on the trigger section will test to make sure the connection is working. It’ll ask you to send a chat message in the group you selected. Once you’ve done it, you’ll get a green success message.

Step 4 – Set an Action

You should see a page with a link that says “Your Zap currently lacks an Action step. Add one now!”. Follow that link.

For this example, we’ll just choose Habitica again and choose the “Create Task” option.

You’ll be asked to choose a Habitica account. Use the one you created in step 3.

Next, you’ll be able to create the template for your new task.

Under type, choose todo.

Under name, put “New Message From ” and then press the insert a field button to the right of the input. A selection of options should appear, pick User Name.

In the notes field, select the insert field button and choose Chat Text.

You can skip the rest of the options for this example. Many pertain to specific task types other than To-Dos.

There will be a test page to make sure that Zapier can successfully create the To-Do. Test the connection and make sure that a new To-Do appears on your Habitica account (you may have to select the sync button).

Step 5 – Test out your zap!

All that’s left to do is try it out. Go to your party page and send a message. Wait a few seconds, go to the task page, and press the sync button. You should see your new task with the details from your chat in the To-Dos column.

Now, every time someone posts a message in your party, you’ll get a new To-Do!

Step 6 – Make something useful

I get it, what we made isn’t particularly useful. But I hope it inspires you to make something that is.

Now that you’ve signed up for the Beta, Habitica will show up in the list of your apps to use when making a new Zap. Here are some ideas for zaps you could make:

  • Whenever you add an event to your Google Calendar, create a new To-Do
  • Whenever a new issue is added to a Github Repo, create a new To-Do
  • Whenever someone follows you on Twitter, score a habit

Got a great idea? Share it in the Aspiring Comrade’s Guild!

PR Process Update

Hello Habiticans! We’re moving our posts from our defunct dev blog, which was hosted at devs.habitica.com, to this blog. Note that much of the information in these posts is outdated, and we preserve them here for historical purposes as well as in case they contain any useful information for posterity. Some posts have been omitted as they will not be relevant for those purposes. Please reach out to us if you’d like to receive a list or document containing a complete record of the devs.habitica.com content.

We’re aware that the amount of time that it takes for us to review a PR has been disappointingly long. That can be discouraging for contributors, and we don’t want that! As a result, we’re making some changes to the ways that we manage PRs to make the process more straightforward.

  1. Effective immediately, SabreCat will be dedicating more of his developer time specifically to reviewing PRs.
  2. PRs for bug fixes will jump to the head of the line, because those are highest priority. Other than that, PRs will usually be reviewed in the order that they were received, though sometimes a high priority PR might skip ahead.
  3. If your PR has a “needs work” label and we haven’t heard from you for a month, we will ping you on GitHub. You’ll have a week to respond to confirm that you’re still working on the PR, or your claim will lapse. You’ll be able to refresh a claim three times before it will lapse automatically. Once your claim has lapsed, we’ll either take over the PR ourselves or release it back to the rest of our dev community. If you completed some work that we decide to use, you’ll receive partial contributor credit. We reserve the right to shorten these turnaround times if the PR is of high priority–we’ll clearly state a due date in those cases.
  4. We also welcome members of our coding community to review each others’ work! GitHub makes it easy for contributors to add comments and suggestions on outstanding PRs. (But please only review if you have a good understanding of the code and experience with the functional area of the PR in question.) We are considering the possibility of having a new contributor title for users who we notice are doing a particularly good job with reviewing and testing PRs. More information will be given later if we decide to proceed with this!
  5. We have an upcoming website redesign that will fix a lot of pending issues. Once those are done, we will do a big sweep and close a lot of tickets. After the redesign is done, some of our other developers will also start allotting time each week to working on PRs, so the process should speed up even more!