Some people eat, sleep and chew gum, I do genealogy and write...

Sunday, December 17, 2017

Integrating Genealogy Into Your Lifestyle

The term "lifestyle" has been used a lot the last few years as the focus of our society, especially here in the United States, has become more "me" oriented. I never could figure out what my own "lifestyle" consisted of. The term lifestyle also seems to go with the term "active retirement" and even the idea of having a "bucket list." All of these concepts seem foreign to me. When I retired from my active law practice (more based on interests rather than economics) I was already so heavily involved in genealogy that I hardly noticed the change. I simply did more writing and more research. I also continued to volunteer at the Mesa FamilySearch Library.

However, I realize, in talking to some of my friends, that the idea of retirement evokes every emotion from anticipation to terror. One of my friends was facing a year-end retirement situation and was at a complete loss as to what he would be doing once he did not come to the office every day. In his case, he was facing a serious "lifestyle" change in the current jargon.

Genealogy can be a total "lifestyle" commitment. I happen to associate with people who, like me, wake up thinking about genealogy and go to bed with the same topic. I am certain that this "lifestyle" has little general attraction and few would look forward to doing something as time intensive and totally adsorbing as genealogy when they finally "retire." In fact, few can fit genealogical research into their current "busy" days even if they are far from retiring.

When the "outside world" thinks of genealogy or family history, they think of a "hobby" or part-time activity that might take a few hours a month, not an all-encompassing activity that looks a lot like a full-time job. True, you could do a little bit of research once and while and consider yourself a genealogist or family historian. But from my perspective, genealogy is a professional level activity. My genealogical activity actually takes me more time and mental effort than my intense legal trial practice.

Is there a middle ground? Can you be a "genealogist" and still have a life outside of genealogy? Of course, the answer is yes. By the way, I am not the best example of a balance between genealogy and other interests, but just because there are "full-time" genealogists does not mean that there is no place for those with less time and inclination.

American Ancestors Opens New Mayflower Passengers Website
With the opening of a new Mayflower Passengers website by American Ancestors, New England Historic Genealogical Society, there will be one more major source for the most reliable information for the ancestors of a significant percentage of the entire U.S. population. Quoting from the emailed announcement:
We are pleased to announce that we recently launched a new interactive website to commemorate the upcoming 400th anniversary of the Mayflower landing. The site presents the most authoritative biographies to date of the Pilgrims who set sail for a new world 397 years ago—available for free for the first time. The biographies are drawn from Robert Charles Anderson’s Pilgrim Migration, the biographical details include information on births, marriage, children, and roles in Plymouth Colony. As we approach 2020, more in-depth features and scholarly material will be added to the site to commemorate the historic Mayflower voyage.
Many of the entries for these individuals, particularly on the Family Tree, have been subject to massive duplication and variation. I commonly receive notice from FamilySearch concerning changes made to my own Mayflower ancestors with long lists of changes. For example, Mayflower Passenger, Francis Cooke, has over 45 changes to his information in the last week before this post was published.

As stated above, the year 2020 will be the 400th anniversary of the arrival of the Mayflower in America. I suggest that a fitting tribute to their memory would be to clean up the entries on the Family Tree and keep them consistent with the information so welcomely being provided by the New England Historic Genealogical Society.

Saturday, December 16, 2017

1 Out of 5 Children in the United States are hungry and genealogy

As I am currently driving across the United States from my home in Provo, Utah to serve a mission for The Church of Jesus Christ of Latter-Day Saints in Maryland, I have been noticing the billboards claiming a huge number of children in the United States suffering from hunger. In the past, I spent years serving in a local charity that feeds the homeless and others who need food. Also, members of my family regularly volunteer in food programs to feed school-age children. The Church is also heavily involved in humanitarian services. See These services are supported by the members' voluntary contributions and fasting from their meals on one Sunday a month.

As a result, I have had very personal interest in the problem of both homelessness and hunger in the United States and elsewhere. But I am also a former trial attorney and a genealogist and therefore I am acutely aware of the need to support anything we say or record with adequate sources. Just as I would not go to court without evidence to support my case, I would not put any information in my family tree that I could not support with documentary historical records.

Now, what about the signs I am seeing along the road? They are simply and easily proved to be false. Finding information to contradict the statements is extremely easy. See Forbes, November 20, 2011, entitled "Are One In Five American Children Hungry?"

Now, unfortunately, the same types of statements are commonly made in online family trees and other genealogical publications. One of the most common is the statement, which I heard quoted again this week, about the popularity of genealogy as either the most popular or perhaps the second most popular hobby in America today. I have posted many times about my efforts to substantiate this claim and have shown over again that it is unsupported by any valid statistics from any source whatsoever.

There are a myriad of programs including school lunches, food stamps, and other similar programs as well as private charities that provide food the hungry, As the Forbes article concludes the greater problem today is juvenile obesity, not hunger. This is not to say that juvenile hunger does not exist in America. But exaggerating the problem does not help cure the situation. Before you contribute to a charity that uses false statistics to support its fundraising, you might investigate other more forthright and deserving charities and churches that are addressing the needs of our children realistically and at the very basic level.

Going back to genealogy, it is imperative that we do not pad our family trees with publically broadcast but unsubstantiated information. If we wish to speculate, do so in privacy and don't publish your speculations online.

Digital Public Library Adds Digital Maine
The Digital Public Library of America or DPLA announced the addition of Digital Maine to their collections. This brings the number of records online on the website to 18,666,818.

As is explained by the DPLA Blog post:
As we prepare to ring in a new year, we are pleased to share the collections of Digital Maine, which joins Oklahoma, Florida, Montana, Maryland, Michigan, and Illinois, as the seventh new partner whose collections have been added to DPLA in 2017. With Maine State Library at the helm, Digital Maine contributes state documents and records, dating back to the Revolutionary War, as well as materials from local libraries and historical societies across the state. 
You’ll find some “classic Maine” materials like rocky coastlines, cold weather, and lobster recipes, but also look for the materials that uniquely represent the state’s many small towns and local communities. For example, this collection of glass plate photographs documents the rural logging town of Monson at the turn of the twentieth century. Photographs and maps from Kittery, Maine’s Rice Public Library and other institutions record the happenings at the Portsmouth Navy Yard, which dates to 1800 and is the Navy’s oldest continually operating shipyard.
As noted, many other states have now linked their digital collections to the centralized searches of the DPLA. The website lists all of the DPLA partners. For genealogists, this is an excellent list of sources for additional information. Many of the records on the DPLA are genealogically valuable. I had heard that the digital Books collection on was to be added as a partner program, but I had not heard anything more since the original announcement.

Wednesday, December 13, 2017

How important is high resolution for scanning and photography?

Are you tempted to join the megapixel race? Are you concerned about the resolution of your digitization efforts for photos, paper records, and other genealogically important documents? Do you use the megapixel count of a camera or smartphone as a factor in your purchase decisions? These issues and more concern anyone trying to digitize records or take photographs. Genealogists and photographers share some of the same concerns.

I have written on this topic several times in the past. Here is a list of some past posts that deal with aspects of this topic:
This list could go on and on. In a recent post, I expressed my views on the challenges of genealogy and I included an issue about the unrealistic digital resolution and file format requirements imposed by those engineers and administrators of online collections thereby increasing inability of the larger collections to ingest smaller collections of records. On reflection, that topic needs more explanation and discussion. 

In response to my post on the challenges to genealogy, I got the following comment:
I have always been a believer that preservation should be performed at the highest possible resolution. As time has passed, as you mention, this could be 50 Megapixels today, and who know how much tomorrow? But the biggest advantage of 50 vs 12 Megapixels is the ability to zoom in and examine details closely. I have found this very helpful with things like scans of old vital records where correct interpretation of handwriting, for example, requires great magnification. It is useless if zooming in only results in a highly pixelated image. This applies likewise to photographs where the only image of GG Grandpa is a tiny section of a larger image. If I want to recognize his features clearly, I am grateful for a 50 Meg scan. Obviously, as you mention, file size (storage capacity) is an issue, but less so as time passes. Therefore, I support the ". . . unrealistic digital resolution and file format requirements imposed by those engineers and administrators of online collections . . .". Tomorrow's researchers will thank us for adhering to those high standards.
Is there a direct relationship with a high megapixel count, say 50 megapixels or more, and the ability to recognize small features in either a photograph or another type of document?

We need to start any discussion of this type with some observations about physical reality.

I will start with photographs. Analog photographs using photographic film are considered to be continuous tone images. However, the resolution of a photograph depends on the type of film used. The sensitivity of film to light is measured in a number assigned by the International Organization for Standardization or ISO or the American Standards Association, now known as the American National Standards Insitute, or ANSI whose standard is usually designated by the older acronym, ASA number. There is a direct relationship between a film's ISO/ASA number and its ability to resolve fine detail, i.e. resolution. The higher the ISO/ASA number, the larger the grains of light-sensitive material, usually some compound of silver, used to capture the image. These numbers are usually used to represent the "speed" of the film or the time it takes to form an image. The higher the numbers, say around 1000 or 2000, mean that the film is very "fast." The tradeoff is always a loss in detail i.e. graininess of the image.

There is no free lunch, greater resolution means smaller discrete light sensitive elements. Photographers know that high ISO/ASA numbers (or fast film) mean a decline in detail in direct proportion to the additional speed. For those wishing to digitally reproduce film photographs, the resolution of the copy cannot exceed the original. Any document or photograph has a certain limit of resolution. Once a duplication method reaches that point of resolution there is no more information in the original that will be lost because of the copy. It may seem counterintuitive, but higher resolution scanning or photography past a certain threshold will simply result in larger file sizes and not any more detail. Once that limit has been reached, there is no more information to obtain.

I am not here talking about photographs of real-life objects, I am talking about copying historical records and photographs, essentially digital reproductions of actual analog documents.

Here is an example of what I mean. This is a microfilmed copy of a record from the website that was previously microfilmed and has now been made available in a digitized copy:

Now, how did this image come to be on the website? In a simplified explanation, someone had access to the original record and then made a photographic copy of the original using some type of microfilm. Here, the resolution was determined by the type of film, probably with a very low ISO/ASA number below 100, i.e. with the highest amount of detail available. Now, to move this image into the digital world, FamilySearch made a digital image at some extremely high resolution (for a digital image) and then processed that image for display on its website. What about the resolution of this image? Well, first of all, it is a JPEG image and we will have to view the image on our computer's monitor. Let's see what happens to this image at magnification. Here is a screenshot of the image at 300%.

Hmm. there appear to be some problems with the original. There is a great deal of bleed through from the back of the page. What about higher resolution? Here it is again at 600%.

Is there an upper limit? Yes, here is the image is again at 800%:

At this point, further magnification will simply start more pixelation and not provide any more detail. Could this be extended indefinitely be making the original with a higher digital pixel count? In reality, the file size would increase dramatically but you would still be limited by the resolution of the original image. Here is the same image at 1200% magnification.

Any higher and the image will start to become unrecognizable. Where can you see the most detail? Guess what? That depends on how closely you look at the image. If you stand some distance back, the high magnification images look just like the ones with lower magnification.

There is a reason why the Libray of Congress established standards as set forth in its "Guidelines: Technical Guidelines for Digitizing Cultural Heritage Materials." There is a balance between increased resolution and the preservation of the detail in a document or photograph. Higher resolutions give you larger file sizes but at some point, no more information from the original.

There is no free lunch. You cannot beat the system and the system is physics.

Tuesday, December 12, 2017

The Ultimate Challenges of Genealogical Access to Digitized Records

Online genealogically important historical records are rapidly transforming the way genealogists find their ancestors and extended ancestral families. Billions of new records are being added every year by the large online genealogy companies. It would seem that this flood of new records could go on indefinitely. But there are strong indications that the flood may soon diminish to a trickle unless the genealogical community can overcome some looming obstacles.

These obstacles to the continued increase in the number of online genealogical records fall into a number of categories that include the following:
  • Political restrictions on the access to records
  • The monetization of records by governments and other organizations
  • The reverse side of the principle of economies of scale, i.e. the cost of digitizing smaller collections of records
  • Unrealistically restrictive copyright and other similar restrictions on historical records
  • The unrealistic digital resolution and file format requirements imposed by those engineers and administrators of online collections thereby increasing inability of the larger collections to ingest smaller collections of records
  • The costs of maintaining ever larger databases including the costs associated with migrating file formats over time
  • The lack of community standards for record formats and the inability of users to move records from one online family tree program to another
  • Ignorance of the members of the genealogical community as to the identity and availability of online digital record collections
Here is my viewpoint on each of these obstacles:

Political restrictions on the access to records

The most difficult and pervasive obstacles to continued digitization are the politically imposed restrictions on record access around the world. In some areas, record access, much less digitization of those records, is virtually impossible. It is clear that the ability of individuals to access records is a major threat to oligarchies and repressive governments no matter what their origin or motivation. This is not an issue that is limited to national governments but can operate on a local level when politicians believe their control and power are threatened by access. In the United States, for example, we would not have national and local freedom of information statutes were politicians and bureaucrats cooperative in providing access to "public" records. In addition, the ongoing destruction of genealogically important records and the attacks on state archives and libraries continues to threaten the availability of records around the country. Absent major changes in some countries of the world and even in parts of less repressive countries, many records will remain unavailable. Ultimately, the reasonably accessible records around the world will all be "cherry picked" leaving huge numbers of records locked up by repressive governments. 

The monetization of records by governments and other organizations

It is a fact of life for genealogists that access to more and more records around the world are being used by those who maintain or archive those records as local revenue streams. This occurs wholesale, even in the United States, for many types of records. For example, in almost every state of the United States of America, if you are born, get married or die and you or your family want a copy of an official government certificate of any of those events, you will have to pay a fee to obtain a copy. In England, it a common practice for local ecclesiastical parishes to charge a fee for access to historical parish registers. I am not of the opinion that all records must be free, but the monetization of the records makes their acquisition by free websites such as very unlikely. It also makes the overall cost of digitizing and making the records available much more expensive.

The reverse side of the principle of economies of scale, i.e. the cost of digitizing smaller collections of records

Record acquisition and digitization are labor intensive and the equipment needed for high-quality images is still quite expensive. For these reasons, extensive record digitization efforts can achieve economies of scale. On the other hand, smaller projects with fewer records require that those same assets but must be used with far fewer records so the cost per record becomes a major concern. In other words, smaller collections have some of the same overhead considerations as larger collections making the cost per record much higher. Also, the logistics of obtaining smaller records are usually about the same as larger collections. The results are that there are distinct disincentives to acquiring smaller collections of valuable records.

Unrealistically restrictive copyright and other similar restrictions on historical records

Unfortunately, US Copyright law is vague and overly restrictive. Current copyright claims will likely be in effect longer and any person now living. Even old copyright claims dating back to the 1920s and 30s will likely be arguably enforceable longer than anyone now living. This could be called the "Mickey Mouse" effect. In both 1976 and 1998, the existing copyright interests were extended for up to 120 years from the year of creation. See the post, "How Mickey Mount Keeps Changing Copyright Law." Because the provisions of these laws are vague, all sorts of claims to copyright now cloud the ability of genealogists to access records online.

In other cases, record repositories claim a "contractual" ownership right to documents that are clearly in the public domain. These claims prevent the free use of all sorts of records, photographs, and other documents. Until there is a realistic overhaul of the copyright laws and a clarification of the unfounded claims by repositories, many valuable records will be subject to restricted access.

The unrealistic digital resolution and file format requirements imposed by those engineers and administrators of online collections thereby increasing inability of the larger collections to ingest smaller collections of records

This particular issue is less obvious than any of the other challenges facing genealogical access to digitized records. Essentially, those who are charged with developing the standards for online digital preservation impose unrealistic restrictions on the process of digitization. For example, we have long known that the highest resolution is approximately the equivalent of 170 dpi or PPI (pixels per inch) when viewed at 20 inches. In contrast, the average laser printer can print at 300 dpi or roughly double the eye's resolution. See "What is the highest resolution humans can distinguish." Presently, some of the digitization efforts going on around the world are using cameras that have up to 50 Megapixel sensors. Most of the documents being digitized could be adequately preserved with a camera of about 12 Megapixels the resolution of a present smartphone. The U.S. Library of Congress has established a publication called "Guidelines: Technical Guidelines for Digitizing Cultural Heritage Materials." Quoting from that publication concerning documents:
Image capture resolutions above 400 ppi may be appropriate for some materials, but imaging at higher resolutions is not required to achieve 4* compliance.
The practical effect of an artificially imposed higher standard is that many smaller collections are going to be lost because the large online genealogy companies refuse to ingest even images at the Library of Congress standard or make the process of obtaining images so complicated as to make smaller collections unfeasible.

The costs of maintaining ever larger databases including the costs of migrating the file formats over time

Even with the dramatic decreases in the cost of memory storage, huge online genealogical collections, especially those with photos, videos and audio files, can eat up huge amounts of memory into the hundreds of Terabytes. Adding in the cost of acquisition and maintenance makes this an extraordinary effort. Adding new records can have an incrementally higher cost. It is only a matter of time until these huge collections run into an economic and practical limit. However, there is a long way to go before this will happen. Right now, there is a major concern with the need to migrate existing collections as new file formats and operating systems evolve. Apple recently introduced a new file format for its smartphones, HEIC, and this will eventually affect the large online genealogy companies.

The lack of community standards for record formats and the inability of users to move records from one online family tree program to another

This is a major issue and I have written about this recently. Without community standards, each of the large online database companies is essentially an island of their own file formats. Without a standard way to exchange data, if one or more of these companies fail, much of their data could be lost.

Ignorance of the members of the genealogical community as to the identity and availability of online digital record collections

Let's face it. There is a constant loss of genealogical data due to genealogists who ignorantly or even intentionally fail to share their data and adequately prepare for its preservation upon their deaths. This attrition of records will always be a drag on preservation efforts.

There is always hope in the future and it is always possible that some or all of these issues will be resolved, but right now they stand as genealogy's greatest challenges. 

Sunday, December 10, 2017

Can your public library help you with your genealogy?
It may not occur to you but your local public library may be an excellent source of information for genealogical research. For example, the Hedberg Public Library in Janesville, Wisconsin has a long list of databases available both for use in the library and online with a library card. Some local public libraries, such as the Allen County Public Library headquartered in Fort Wayne, Indiana has one of the most extensive genealogical collections in the United States.

Here is a screenshot of the Allen County Public Library Genealogy Center website.

Your local library may be sponsored by your town, city, or county or all three. In Mesa, Arizona where I lived for many years, we had an excellent local Mesa Public Library. We also had an excellent county library system, the Maricopa County Public Library System, and a State Library in Phoenix. We also had an extensive system of Family History Centers around the Salt River Valley including the one where I was a volunteer, the Mesa FamilySearch Library.

It was interesting to me that many of the people I met in the Phoenix area who professed to be interested in genealogical research had never visited the Mesa FamilySearch Library and some had not even heard of its existence. There are over 5000 Family History Centers around the world and it is likely that there is one near you. See the Get Help menu for a location near you.

Sometimes we tend to judge a library by whether or not it has a particular book or other items we are searching for. But libraries can be surprising in the resources they have in their collections. If you are going to travel to an area where your family lived to do research, take the time to contact a local library in the area and ask about their resources.