Photo credit: Stephen M. Scott
 +   -  text size:

Blog

All entries categorized “data-management”


Importing Text Files with Variable Names to R

Wednesday, Nov. 26th, 2014 10:47a.m.

I am attempting to learn R. This is either a great thing or a terrible, terrible mistake while on the tenure clock. But, all the cool kids are doing -- so even though they might also jump off a bridge, I'm going to jump into R.

The hardest part so far is doing things that now come as second nature to me in Stata. Although R's tools are much better in the long-run, learning what types of objects different functions return and such ends up being a very high learning curve.

A while back (all posts are a while back now), I wrote a post describing how to import data with variable labels in Stata. The idea was that I could keep the variable labels with the data in an text file so that I could always figure out what the variables were. It doesn't require an extra codebook or additional files that could be lost.

I have now replicated that script for R, and it is below:

  tags: data-management, R, stata-to-R category: Programming

Nesting Stata Macros, or Hacking a Hash Map

Monday, June 6th, 2011 6:37p.m.

Programming in Stata is relatively straightforward and this is partly because the programming syntax is both powerful and relatively straightforward. There are, however, a few minor annoyances in Stata's language including using the backtick and apostrophe to indicate local macros (i.e.,`localname'). Among these shortcomings, I would argue that the lack of anything like a list in Stata's language is one of the largest.

In most langauges, you can store a list of items and refer to the item in the list by some sort of index. This is particularly helpful for iterating over the same step multiple times. Lists generally come in two flavors: lists to which you can refer to an item by its position in the list or lists which you can refer to by a keyword (called hash maps in computer science lingo). Stata's matrices can be used for the first, though doing so might become complicated if you want to do something besides storing basic numbers or strings.

  tags: data-management, macros, Stata, tips-n-tricks category: Programming

Structuring Work: Data Cleaning and Construction, Laying the Foundation

Saturday, April 16th, 2011 11:37a.m.

In the last step, we downloaded all of our data and deposited into directories that store this source data, backed it up, and write-protected the files. Now that we have done all of that, it is time to start working with the data! There is only one problem: almost inevitably, the data do not come neat, tidy, and ready to use. Often, the data contain major problems and need to be constructed in order to be usable. In this installment, I will write about managing files for cleaning, constructing and storing datasets.

  tags: advice, data-management, research-process, workflow category: Structuring Work

Structuring Work: Data, The Foundation of Work

Monday, March 14th, 2011 3:50p.m.

After establishing where my root directory resides resides, it is time to actually get to work. As with any endeavor, success begins by laying a solid foundation and with academic work that begins foundation is our data.

The most fundamental skill to academic success is asking good questions and acquiring data to answer those questions. Yet, in quantitative research, that skill is useless without the ability to manipulate data into useful formats that are capable of answering the good questions. Data cleaning, construction, and manipulation constitute well over half of my work on major quantitative projects.

  tags: advice, data-management, research-process, workflow category: Structuring Work

Structuring Work: The Root, Where it all Begins

Friday, Feb. 11th, 2011 1:02p.m.

In my last post, I explained the value of a directory structure: consistent file management structures a disciplined workflow that increases productivity. The magnitude of its importance was a revelation that occurred largely after graduate school as the result of starting a new job.

When I moved to start my new job, I needed to move my files to my new computer. In transferring my files, I realized that my work that followed my well-defined workflow transfered easily, while the work that didn't follow the workflow did not.

The contrast between the ease with which I started the well-structured work and difficulty getting up to speed on disorganized pieces threw in sharp relief the importance of maintaining a workflow structured by a consistent file management system. For those well-organized projects the only difference being on my new computer was that I began work from a different "root directory".

  tags: advice, data-management, research-process, workflow category: Structuring Work

Structuring Work

Friday, Feb. 4th, 2011 10:04a.m.

When I say that one of the most important things that I did in graduate school was set up a directory structure and workflow for my files, I am not kidding. Reading theory, learning statistical methods, and writing literature reviews were all important. However, just as important -- though not nearly as sexy -- is setting up a file structure and working directory.

Despite how trivial it sounds, maintaining a well-designed directory structure not only provides a framework for files, it structures productive work.

Given how important it was for me, I will attempt to explain the directory structure that I developed. Let me begin by saying that I am not an expert at developing directory structures. There are experts in these matters. Though I had an interest in becoming an expert at file management, I was too busy trying to become an expert in what I was actually studying to have the time. I will lay out in an ongoing series of posts the basic intuition behind my posts, what has seemed to work (and not) with this system, and improvements I would like to make. I would, of course, be interested in feedback and or comparisons to what others do.

  tags: advice, data-management, research-process, workflow category: Structuring Work

Importing Text Files with Variable Names to Stata

Friday, July 23rd, 2010 1:17p.m.

I have come across a problem several times that has been relatively frustrating to deal with. I have data that is downloaded from a site (specifically the Census (which is why this comes up consistently) in which the first two lines of the data contain the variable name and variable description respectively. This is incredibly useful for documenting data. Rather than attempting to figure out what variable pct001001 means, the description of the variable is right there.

The problem with data in this format is that Stata imports variables as string variables with the first observation being the variable description. I could pull the first two lines of the data out of the original dataset, transpose the rows and columns, save them in a separate text file, and then import the variable names and descriptions. However, managing two files means that it is more likely that one gets lost or I forget to send one of the files to a colleague working on the paper, or any number of other problems that could be experienced by separating these two files. Having one single file would be far superior and that is what the code below is designed to accommodate.

Data available from the U.S. Census comes in the following format (data is clipped):

  tags: data-management, Stata, strings category: Programming

Front Page

About

  • Information about the purpose and topics of this blog can be found here.

Feeds

Archive

Categories

Tags

Miscellany

  • The views presented here are solely and entirely my own, they do not represent those of my colleagues, employer, or any funding agencies which may support me.
  • The writing on this blog is covered by a Creative Commons License (described here). Feel free to distribute or re-post with a link to the original content provided that it is freely available to others.
  • Creative Commons License