Exporting from Hipchat Server or Data Center for Data Portability
If you use the Migration Utility to export data your Hipchat Server or Data Center instance, you can also use the migration export format for archive purposes, and, if necessary, you can use a decrypted artifact to re-create your Hipchat history on another platform.
This article explains how to use the Hipchat Data Center migration tool. This tool exports data from Hipchat Server version 2.2.3 or later, and Hipchat Data Center 3.1.4 or later.
On this page:
Use the Migrate Utility to export your data
If you want to export all of your Hipchat history, including chat messages, files and attachments, and other information, use the instructions on the How to migrate from Hipchat Server to Hipchat Data Center page to generate the export artifact. You can skip the importing step.
What is exported
Exported |
Not exported |
---|---|
|
|
Check your Hipchat version
This tool exports data from Hipchat Server version 2.2.3 or later, and Hipchat Data Center 3.1.4 or later only. You may need to upgrade your Hipchat instance to proceed.
Check your Hipchat utilization
Before you start the migration process, you should make sure you know how many files, and how many messages you have in your Hipchat instance. The migration tool requires that the Hipchat instance it runs on have enough memory to process thousands of messages and gigabytes of files.
In general:
- For every 1000 messages, the migration tool consumes an additional 2.6 MB of memory.
- For every 1GB of files, the migration tool consumes an additional 500 KB of memory.
Once you know your file store size and how many messages you have, you can calculate any additional memory requirements. You might need to provision your Hipchat instance with more memory to complete the migration tool run. Once the tool has run, you can reduce the instance's memory back to the original amount, if needed.
(Note that this tool runs an all-or-nothing export. If you continue to use your Hipchat instance after running the export phase of the migration, you will naturally have some chat history that is not included in the export.)
Install the migration utility
Download the migration utility to your local computer: https://s3.amazonaws.com/hipchat-server-stable/utils/migrate/hc-migrate
v3.1.5, last updated July 3rd 2018. SHA512 sum:you may need to scroll --> 818bfbbfc7e8c8e868282a03662c76da257bdcf3395d708192386d06ab00aacdf9a5b3416faa95f30e2446cdec650ea3be8f5f0e36ec725384b7a59a4fa30f0d
Copy the migration utility to the Hipchat instance that you're going to export from. If you're exporting from Hipchat Data Center, copy the utility to any one of the Hipchat nodes in the cluster.
scp -i /path/to/sshkey.pem hc-migrate admin@<hipchat.example.com>:/home/admin/hc-migrate
If you're exporting from a Hipchat Data Center deployment, you may need to copy the migration utility to your cluster's jumpbox (or management portal) then copy it to the node.
Export from Hipchat Server or HipChat Data Center
Reminder!
This tool exports data from Hipchat Server version 2.2.3 or Hipchat Data Center version 3.1.4 or later. You may need to upgrade to a compatible version of Hipchat to proceed.
Connect to the Hipchat node's command-line interface using a terminal or SSH.
ssh -i /path/to/sshkey.pem admin@<hipchat.example.com>
Escalate your user permissions to root.
sudo dont-blame-hipchat
Switch to the directory containing the migration utility.
cd /home/admin/
Update the permissions for the migration utility so it can be run.
chmod +x /home/admin/hc-migrate
Verify the checksum and the version.
sha512sum hc-migrate /home/admin/hc-migrate -version Hipchat Migration Utility version 3.1.5 (2018-07-03)
Retrieve the password to the source Hipchat database. You'll use this (without the quotation marks) in the next step. For HipChat Server, use the following command:
cat /hipchat/config/site.json | jq -r .databases.hipchat.pass
For HipChat Data Center, use the following command:
cat /hipchat/config/site.json | jq -r .databases.hipchat_postgres.pass
Copy, edit, and run the following example command to start the export for HipChat Server:
By default, the export artifact is saved in the directory from which you run the command. You can use the the
--export
flag to specify another location./home/admin/hc-migrate --export "HCS_EXPORT_NAME.tar.gz.aes" --aes_password "ARCHIVE_PASSWORD" \ --db_password "DB_PASSWORD" --verbose |& tee exportHipchat.log
For HipChat Data Center, use the following command:
/home/admin/hc-migrate --hcdc --export "HCDC_EXPORT_NAME.tar.gz.aes" --aes_password "ARCHIVE_PASSWORD" --postgres_url "dbname=PSQL_DB_NAME host=PSQL_HOST_URL user=PSQL_DB_USER password=PSQL_DB_PASS port=5432 sslmode=disable" --redis_url "REDIS_URL" --redis_password "REDIS_PASS" --verbose |& tee /home/admin/migrate.log
A sample command would look something like this for HipChat Server:
/home/admin/hc-migrate --export "/home/admin/20171115_migrate.tar.gz.aes" --aes_password "hipchat" --db_password "reallysecurepassword" --verbose |& tee /home/admin/exportHipchat.log
A sample command would look something like this for HipChat Data Center:
/home/admin/hc-migrate --hcdc --export "hcdc_export.tar.gz.aes" --aes_password "password" --postgres_url "dbname=hipchat_postgres host=kdmbphm7asp2ad.cveoy2l56qjq.us-east-1.rds.amazonaws.com user=hipchat password=ChangeMe123! port=5432 sslmode=disable" --redis_url --"12345.6789.ng.0001.use1.cache.amazonaws.com:6379" --redis_password "my_redis_password" --verbose |& tee /home/admin/migrate.log
When the export is complete the logs will stop in the terminal and the command will exit. The end of the log output for a successful export looks like this:
DEBUG 2017/11/15 01:59:04 Writing rooms/711/history.json DEBUG 2017/11/15 01:59:04 Writing rooms/712/history.json DEBUG 2017/11/15 01:59:04 Writing rooms/713/history.json DEBUG 2017/11/15 01:59:04 Writing rooms/714/history.json DEBUG 2017/11/15 01:59:04 Writing rooms/715/history.json DEBUG 2017/11/15 01:59:04 ======== *** Export completed successfully! *** ========
You can now copy the .tar
file to another machine to work with the exported archive.
Decrypt the Export Artifact
The migrate utility creates a .tar
file (tarball) which is encrypted at generation time using a password you set. Before you do anything with the file, you'll need to decrypt it.
You can decrypt the Hipchat archive by running the following command, substituting the password you provided to the migrate
command for my_password.
openssl aes-256-cbc -d -k my_password -in my_export.tar.gz.aes -out my_export.tar.gz
Open the Export Artifact
Next, you can run the tar
command to extract the contents of the tarball.
tar xzf my_export.tar.gz
You will need to re-encrypt the Artifact if importing again to HipChat using the hc-migrate tool.
Export contents and file system
This information is provided to help you understand how each type of data is exported, and how it is structured in the export format. You can use this information to help write a parser, which you might use to copy the exported data to another system or for archive and retention purposes.
This documentation only applies to archives created by the Migration Utility. The export command line and admin UI produce different format that is documented elsewhere.
├── autojoins.json
├── emoticons.json
├── files
│ └── img
│ └── emoticons
│ └── 1
│ ├── hcdc-1522683674@2x.jpg
│ ├── hcdc-1522683674@3x.jpg
│ ├── hcdc-1522683674@4x.jpg
│ └── hcdc-1522683674.jpg
├── metadata.json
├── preferences.json
├── rooms
│ └── 1
│ ├── files
│ │ └── BcudLU14VxUEeUK
│ │ ├── img.JPG
│ │ └── img_thumb.jpg
│ └── history.json
├── rooms.json
├── users
│ ├── 1
│ │ ├── avatars
│ │ │ ├── PrypXtu4rN8sEFL_125.jpg
│ │ │ ├── PrypXtu4rN8sEFL_36.jpg
│ │ │ └── PrypXtu4rN8sEFL.jpg
│ │ └── history.json
│ ├── 2
│ │ └── history.json
│ └── files
│ └── 0Lp9rkaHfzLpPow
│ ├── img.JPG
│ └── img_thumb.jpg
└── users.json
Auto-Joins
autojoins.json
contains a list of autojoin preferences for users.
id
- string - format is:"pref:autoJoin:userId"
data
- list of json objects
jid
- string - JID of the room/conversation that the user should auto joinname
- string - display name of the room/conversation
{
"AutoJoins": [
{
"id": "pref:autoJoin:1",
"data": "[{\"jid\":\"1_default@conf.btf.hipchat.com\",\"name\":\"Default\"},{\"jid\":\"1_1@chat.btf.hipchat.com\",\"name\":\"HCDC User\"}]"
},
...
}
Emoticons
emoticons.json
contains references to all custom emoticons uploaded by users.
ID
- integer - emoticon identifieruser_id
- integer - reference to user that uploaded emoticongroup_id
- integer - group identifier this emoticon belongs to (should always be 1)path
- string - corresponds to the location of the file in the export archive relative tofiles/img/emoticons
shortcut
- string - this is what you type to use the emoticon in chat (for example typing(hcdc)
displays the hcdc emoticon)width
- integer - width of standard emoticonheight
- integer - height of standard emoticonaudio_path
- string - always empty - deprecated, used by legacy clients only
Note: The export archive contains different sizes for the emoticon for different devices, and includes the context in which it is used (in text vs. a single emoticon).
{
"Emoticons": [
{
"ID": 217,
"user_id": 1,
"group_id": 1,
"path": "1/hcdc-1522683674.jpg",
"shortcut": "hcdc",
"width": 28,
"height": 28,
"audio_path": ""
}
]
}
Rooms
rooms.json
contains details about the rooms from your Hipchat instance.
id
- integer - room identifiercreated
- string (ISO 8601) - date room was createdguest_access_url
- string - ifnull
, guest access was disabled for this room, otherwise this contains an URLis_archived
- boolean - true if the room was archivedmembers
- array of integers - list of integers correspond to user IDscanonical_name
- string - used for the room JIDname
- string - display name for the roomowner
- id - user id of room creatorparticipants
- array of integers - (legacy) integers correspond to user IDsprivacy
- string - public or privateroom_admins
- array of integers - user IDs of room adminstopic
- string - room topicis_deleted
- boolean - true if the room was deleted
[
{
"Room": {
"id": 1,
"created": "2018-04-02T15:31:34+00:00",
"guest_access_url": null,
"is_archived": false,
"members": [],
"canonical_name": "default",
"name": "Default",
"owner": 1,
"participants": [],
"privacy": "public",
"room_admins": [
1
],
"topic": "",
"is_deleted": false
}
}
]
Users
users.json
contains details about the users from your Hipchat DataCenter instance.
id
- integer - user identifiername
- string - user's display namemention_name
- string - user's@mention_name
to be referenced in chatsemail
- string - user's emailtitle
- string - the user's job title as displayed in the HipChat GUIavatar
- string - the path alwaysfiles/photos/<userId>/<avatarFile>
and resolves tousers/<userId>/avatars/<avatarFile>.
The export file contains several versions of the avatar at different sizes, with a suffix noting the width.created
- string (ISO 8601) - date user was createdis_deleted
- boolean - whether the user has been deleted or nottimezone
- string - timezone of where user is basedaccount_type
- string -admin
,user
, orguest
details whether the user is a system admin, regular user, or a guest userroles
- enum - list of all roles the user possesses. Will always contain "user". (Additional options are "owner", "admin", "moderator".)rooms
- array - deprecated legacy format, always emptymetadata
- json quoted string - additional information about the user
[
{
"User": {
"id": 1,
"name": "HCDC User",
"mention_name": "HCDCUser",
"email": "user@example.com",
"title": "",
"avatar": "files/photos/1/PrypXtu4rN8sEFL.jpg",
"created": "2018-04-02T15:31:34+00:00",
"is_deleted": false,
"timezone": "UTC",
"account_type": "admin",
"roles": [
"user",
"admin",
"owner"
],
"rooms": [],
"metadata": "{\"confirmation_unix_time\": 1522683094}"
}
},
...
]
User History
The users
folder contains subdirectories for each user id that contain a history.json
folder. The history folder contains a history of messages to other users in the form of an array of PrivateUserMessage
s.
id
- string - uuid of the messageattachment
- object ornull
name
- string - name of attachmentpath
- string - path of the attachment relative tousers/files
in the export archivesize
- integer - size in bytesurl
- string - deprecated legacy format, always emptycreated
- string (ISO 8601) - UTCsenderid
- integer - reference to user identifier who uploaded the filethumbnail
- string - path of the preview image relative tousers/files
in the export archive
attachment_path
- string ornull
- path of the attachment relative tousers/files
in the export archivementions
- array - deprecated legacy format, always emptymessage
- string - message bodyreceiver
- objectid
- integer - reference to the user identifier of the user who the message was sent tomention_name
- string - blank: legacy format, as the export refer to users by ID numbername
- string - blank: legacy format, as the export refer to users by ID numberversion
- string - deprecated legacy format, always empty
sender
- objectid
- integer - reference to the user identifier of the user who sent the messagemention_name
- string - blank: legacy format, as the export refer to users by ID numbername
- string - blank: legacy format, as the export refer to users by ID numberversion
- string - deprecated legacy format, always empty
timestamp
- string (ISO 8601) w/ appended microseconds - time the message was senttype
- string - always"private"
deleted
- optional object - set if message was deleted, note:message
will be empty string.user_id
- integer - the user identifier who deleted the messagedate
- string (ISO 8601) - UTC - when message was deleted, never contains microsecondsreason
- alwaysnull
[
{
"PrivateUserMessage": {
"id": "518a7891-9e13-4a85-9bdb-839e5408e56b",
"attachment": null,
"attachment_path": null,
"mentions": [],
"message": "have you heard of https://google.com ?",
"receiver": {
"id": 1,
"mention_name": "",
"name": "",
"version": ""
},
"sender": {
"id": 2,
"mention_name": "",
"name": "",
"version": ""
},
"timestamp": "2018-04-02T14:54:04Z 777248",
"type": "private"
}
},
{
"PrivateUserMessage": {
"id": "cc48cd3d-7d3a-43af-b4e2-02355dad56cd",
"attachment": {
"name": "0Lp9rkaHfzLpPow/img.JPG",
"path": "0Lp9rkaHfzLpPow/img.JPG",
"size": 3368233,
"url": "",
"created": "2018-04-02T15:43:42Z",
"senderid": 2,
"thumbnail": "0Lp9rkaHfzLpPow/img_thumb.jpg"
},
"attachment_path": "0Lp9rkaHfzLpPow/img.JPG",
"mentions": [],
"message": "nice picture!: https://hipchat.example.com/files/1/2/0Lp9rkaHfzLpPow/GOPR4259.JPG",
"receiver": {
"id": 1,
"mention_name": "",
"name": "",
"version": ""
},
"sender": {
"id": 2,
"mention_name": "",
"name": "",
"version": ""
},
"timestamp": "2018-04-02T15:43:42Z 970263",
"type": "private"
}
},
...
]
Room History
Room history can be found in the rooms
, under the subdirectory named by the id
of the room in the history.json
file. The contents are an array of messages. Messages can be UserMessage
, ArchiveRoomMessage
, GuestAccessMessage
, TopicRoomMessage
, or NotificationMessage
with varying type
fields. Types can be muc_message
(UserAccessMessage
or NotificationMessage
), muc_archive
, muc_guest_access
, or muc_topic
. Each type is outlined below.
All messages contain the following fields:
- id - string - uuid of message - if 0, the message is from an integration (add-on)
type
- string -muc_message
sender
- objectid
- integer - reference to the user identifier of the user who sent the messagemention_name
- deprecated legacy format, always emptyname
- string - user's display name (chosen by the user), if from a user, otherwise shows the integration or add-on nameversion
- string - deprecated legacy format, always empty
message
- string - message body displayed in roomtimestamp
- string (ISO 8601) w/ appended microseconds - time the message was sentdeleted
- optional object - set if message was deleted, note:message
will be empty string.user_id
- integer - the user identifier who deleted the messagedate
- string (ISO 8601) - UTC - when message was deleted, never contains microsecondsreason
- alwaysnull
UserMessage - Room History
A UserMessage
refers to the standard messages sent from users to a room.
attachment
- object ornull
name
- string - name of attachmentpath
- string - path of the attachment relative tousers/files
in the export archivesize
- integer - size of the attachment in bytesurl
- string - deprecated legacy format, always emptycreated
- string (ISO 8601) - UTCsenderid
- integer - reference to user identifier who uploaded the filethumbnail
- string - path of the preview image relative tousers/files
in the export archive
metadata
- array of objects - metadata can vary but generally includes additional information used to display a message, especially with third-party plugins (imgur, twitter, etc.)
[
{
"UserMessage": {
"id": "2088ec9e-dd0d-4214-8f74-45e2fb85109c",
"type": "muc_message",
"sender": {
"id": 1,
"mention_name": "",
"name": "HCDC User",
"version": ""
},
"message": "anyone here?",
"timestamp": "2018-04-02T15:40:44Z 942827"
}
},
{
"UserMessage": {
"id": "6f0c7db4-ad21-4eeb-8f1a-cbcdda84b066",
"type": "muc_message",
"sender": {
"id": 1,
"mention_name": "",
"name": "HCDC User",
"version": ""
},
"message": "https://upload.wikimedia.org/wikipedia/en/thumb/8/89/HipChat_Logo.svg/1200px-HipChat_Logo.svg.png",
"timestamp": "2018-04-02T15:41:32Z 133009",
"metadata": [
{
"url": "https://upload.wikimedia.org/wikipedia/en/thumb/8/89/HipChat_Logo.svg/1200px-HipChat_Logo.svg.png",
"image": "https://upload.wikimedia.org/wikipedia/en/thumb/8/89/HipChat_Logo.svg/1200px-HipChat_Logo.svg.png",
"type": "image",
"name": ""
}
]
}
},
{
"UserMessage": {
"id": "500a3362-c904-4fcb-a5a6-230c618bbd76",
"type": "muc_message",
"sender": {
"id": 2,
"mention_name": "",
"name": "Other User",
"version": ""
},
"message": "File uploaded: https://hipchat.example.com/files/1/1/BcudLU14VxUEeUK/GOPR4259.JPG",
"timestamp": "2018-04-02T15:41:41Z 865877",
"attachment": {
"name": "BcudLU14VxUEeUK/GOPR4259.JPG",
"path": "BcudLU14VxUEeUK/GOPR4259.JPG",
"size": 3368233,
"url": "",
"created": "2018-04-02T15:41:41Z",
"senderid": 2,
"thumbnail": "BcudLU14VxUEeUK/GOPR4259_thumb.jpg"
}
}
},
{
"UserMessage": {
"id": "500a3362-c904-4fcb-a5a6-230c618bbd76",
"type": "muc_message",
"sender": {
"id": 2,
"mention_name": "",
"name": "Other User",
"version": ""
},
"message": "File uploaded: https://hipchat.example.com/files/1/1/BcudLU14VxUEeUK/GOPR4259.JPG",
"timestamp": "2018-04-02T15:41:41Z 865877",
"attachment": {
"name": "BcudLU14VxUEeUK/GOPR4259.JPG",
"path": "BcudLU14VxUEeUK/GOPR4259.JPG",
"size": 3368233,
"url": "",
"created": "2018-04-02T15:41:41Z",
"senderid": 2,
"thumbnail": "BcudLU14VxUEeUK/GOPR4259_thumb.jpg"
}
}
}, ...
]
ArchiveRoomMessage - Room History
A ArchiveRoomMessage
is a message that notifies whether the room has been archived or not. Rooms can be archived and unarchived. The last sent muc_archive message will dictate the status of the room.
is_archived
- boolean - whether or not the room is archived
[
{
"ArchiveRoomMessage": {
"id": "2088ec9e-dd0d-4214-8f74-45e2fb85109c",
"type": "muc_archive",
"sender": {
"id": 1,
"mention_name": "",
"name": "HCDC User",
"version": ""
},
"message": "",
"timestamp": "2018-04-02T15:40:44Z 942827",
"is_archived": true
}
},
...
]
GuestAccessMessage - Room History
A GuestAccessMessage
is a message that controls guest access to this room. This can be enabled and disabled. The last sent muc_guest_access
message will dictate the status of the room.
enable_guest_access
- boolean - whether or not the room allows guest accessurl
- string - the invite url for guests. This includes an 8 character key that's included in the metadata for any guest users.
[
{
"GuestAccessMessage": {
"id": "2088ec9e-dd0d-4214-8f74-45e2fb85109c",
"type": "muc_guest_access",
"sender": {
"id": 1,
"mention_name": "",
"name": "HCDC User",
"version": ""
},
"message": "",
"timestamp": "2018-04-02T15:40:44Z 942827",
"enable_guest_access": true,
"url": "https://hipchat.example.com/gQ82tnQz5"
}
},
...
]
TopicRoomMessage
A TopicRoomMessage
is a message that shows the current topic of the room. This can be updated anytime. The last sent muc_topic
shows the current topic of the room.
topic
- string - the topic of the room
[
{
"TopicRoomMessage": {
"id": "2088ec9e-dd0d-4214-8f74-45e2fb85109c",
"type": "muc_topic",
"sender": {
"id": 1,
"mention_name": "",
"name": "HCDC User",
"version": ""
},
"message": "",
"timestamp": "2018-04-02T15:40:44Z 942827",
"topic": "this is the purpose of the room"
}
},
...
]
NotificationMessage
A NotificationMessage
is a message that dictates a notification displayed in the room as a "card" which defines a specially displayed message.
format
- string - the type of card to be displayedcolor
- string - the color of the card, one ofyellow
,green
,red
,purple
,gray
card
- string - contains an escaped object containing all details of the card to be displayed
[
{
"NotificationMessage": {
"id": "26f32141-8e44-443b-92fa-2d7a97dbb4ac",
"type": "notification",
"sender": {
"id": 1,
"mention_name": "",
"name": "HCDC User",
"version": ""
},
"message": "Card \u003cb\u003eSample link card\u003c/b\u003e",
"timestamp": "2017-11-07T13:53:53Z 634688",
"format": "html",
"color": "yellow",
"card": "{\"style\": \"link\", \"description\": \"This is some information about the link shared.\\nin 2 lines of text\", \"title\": \"Sample link card\", \"url\": \"http://www.website.com/some-article\", \"thumbnail\": {\"url\": \"http://bit.ly/1TmKuKQ\", \"width\": 1193, \"url@2x\": \"http://bit.ly/1TmKuKQ\", \"height\": 564}, \"date\": 1453867674631, \"validation\": {\"safehtmls\": [\"activity.html\"], \"safeurls\": [\"url\", \"images.image\", \"images.image-small\", \"images.image-big\", \"icon.url\", \"icon.url@2x\", \"icon\", \"thumbnail.url@2x\", \"thumbnail.url\"]}, \"id\": \"c253adc6-11fa-4941-ae26-7180d67e814a\", \"icon\": {\"url\": \"http://bit.ly/1Qrfs1M\"}}"
}
},
...
]