You can’t do records management in SharePoint
Let’s start this admittedly provocative post with a question: Anybody out there actually doing records management in SharePoint?
And before you answer, let me emphasize that I mean real records management, like, with actual, system-enabled automated disposition according to your retention schedule.
If you answered “yes” to this question, please jump immediately to the comments section and let us all know (and while you’re at it, give us some indication of how on earth you’re doing it), because based on my experience, I’d be willing to bet the answer to this question is going to be “no” in 99.9% of all cases.
And while I’m in a betting mood, I’d also be willing to bet that if you answered “yes”, you 100% aren’t doing it with out of the box SharePoint, because out of the box SharePoint can’t do records management at the level the vast majority of organizations require—it just doesn’t, people, no matter how much Microsoft claims that it does, or trumpets that fact that they themselves use it to. But don’t just take my word for it, ask Bruce Miller.
Gauntlet – thrown
And lest you think I’m heaping blame on already blame-laden records managers out there, let me say right off the bat that the real problem with records management in SharePoint is that Microsoft doesn’t really understand records management or what it takes to enable it with an enterprise content management (ECM) system.
Exhibit A: SharePoint 2007, which was supposed to be “records management ready”, but required users to put all their records in a separate area (the records center) to manage them. Getting users to work in two different sites, one for records and one for the rest of their stuff? Say it with me: never, gonna, happen. And never did happen.
Exhibit B: SharePoint 2010 didn’t get much better. True, you could now manage records in place, but 2010 drove records retention and disposition using content types, which, if you had a few hundred of them, were incredibly cumbersome to work with from an administrative and architectural perspective; which is precisely how many most organizations need at a minimum to support their retention schedules; which is why I haven’t seen a single organization using SharePoint 2010 to do real records management (i.e., with actual, system-enabled retention and disposition according to the retention schedule), and I run into a few hundred Fortune 1000 companies a year between projects, sales calls, and events.
The problem is bigger than SharePoint
Now before I get accused of bashing SharePoint here, let me let you all in on a dirty little secret: 99% of organizations are not doing automated records management on an enterprise scale. And this is true whether they have SharePoint, IBM FileNet/P8/CMOD, EMC Documentum, OpenText, Hyland OnBase—whatever. In the end, it doesn’t matter: for a whole host of reasons, some of them technical, some of them organizational, so few folks are actually doing automated records management that we could accurately say that no one is.
Let’s look at some anecdotal evidence for my claim and then turn to some of the reasons why before exploring how we might turn this around.
First, at every ARMA, AIIM, EFM, or other ECM-related event I speak at, I always ask the audience, “who’s doing retention and disposition on electronically stored information (ESI) according to the retention schedule?”, and I never get more than a hand or two in response…and then I ask a follow-up: “Even on shared drives and hard drives?”, and all the hands go down.
Second, no client I’ve ever worked with (somewhere north of 120, all Fortune 1000) has managed to do automated records management on any significant basis. And if you widen the sample to all the Fortune 1000 clients my firm has worked with (somewhere between 400-500), there is only one, a North American bank, that is doing so.
In addition to this lone firm, I had one person speak up one time at an event to say that their firm (a global construction company) was doing automated records management on all ESI…and she stuck to her guns even after my follow-up question about all ESI.
So that’s two organizations out of 400-500 my firm has directly worked with, which is about a half a percent—not good numbers considering we’ve had automated records management software in some form since at least the turn of the millennium. And the number gets worse if you include the 1000+ firms I’ve informally polled at events or talked to as part of the sales cycle.
The final word
So SharePoint or no, we are terrible at automated records management across the board, irrespective of industry or platform—I think that’s an indisputable fact. If you disagree, by all means jump in and let’s get a good argument going.
In the next post, I’ll move on from the problem to a discussion of why we might find ourselves in this mess and suggest some ways that we could move forward–ways that some forward-leaning organizations are already exploring to fix their records management problems.
Trackbacks
- All ECM is the same these days – so throw a dart already and get on with it | agile ramblings
- You can’t do records management in SharePoint (part 2) — but you’re probably not doing it anyway | agile ramblings
- You can’t do records management in SharePoint | agile ramblings | Brooks-Budhoo
- Does SharePoint Manage Records? »
agile ramblings 
This, like most everything else, boils down to execution and persistence. On the subject of RM in SharePoint I’ll cop to using FileNet’s web parts to make it “look” to the user like they’re working in SP, but the fact of the matter is the content is under FNET control and that is where I manage and effect disposition. NO ONE does RM on file shares, yup, that’s a fact, but here I use another toy – IBM Content Collector – to slurp out of those locations and ingest into P8 so, again, I can manage the content, including the RM piece. “All ESI,” or at least the attempt to do so, I have to use about six different mechanisms to cast as broad a net as possible and now you’ve inspired me to delineate what they are, how they work, why I use them in a series of posts on my own blog.
Thanks, Joe
Cheers, Pat
Pat,
Thanks for jumping in and sharing your FileNet and SharePoint records experience with the group.
I know lots of folks who are doing some flavor of what you describe, i.e., pulling records into an ECM repository and then (theoretically) managing them as records there. I say “theoretically” because I don’t see too many folks actually managing disposition in an automated way with tools like IBM’s IER (or the equivalent from OpenText, EMC, Hyland, etc.). Maybe for select record types, i.e., those with date- rather than event-based triggers, but certainly not for a significant portion of their retention schedule.
Do you have clients who are successfully automating the disposition of the majority of their record types in FileNet? If so, it would be great to hear more about it (whatever you can share here)…I know folks would appreciate the real-world perspective on the issue.
Cheers,
Joe
Re: “Do you have clients who are successfully automating the disposition of the majority of their record types in FileNet?” The red umbrella people in Hartford do ALL of their disposition with IER, and they do a LOT. They’re very good on both the operational, business side and the technical, administrative side.
Cheers, Pat
Pat,
A question about the folks in Hartford: how are they addressing event-based triggers? Are they building the logic to handle them, or are they reframing how they think about the recordkeeping requirements to make the trigger actionable?
Cheers,
Joe
IER is a very robust platform, OOTB it comes with multiple triggers (internal, external, recurring), phase actions and several workflows. Can pretty much do whatever I want in terms of disposition as long as I define the file plan correctly. The platform’s capabilities do make you think about now only how you can do it, but how you SHOULD do it. That is, if you want to do it right.
Pat,
But how do they handle requirements like “death of the insured + 7” or “termination of the policy + 3”? Are they letting FileNet know about the death of the insured or termination of the policy, or simply setting up a date-based rule to meet the spirit of the law, e.g., “policy origination date + 100”? In my experience, figuring out how to solve for event-based retention is one of the biggest challenges…not only from a technical perspective, but from an ROI/TCO perspective, i.e., is it more expensive to build the logic to enforce the event-based trigger or to over-retain based on a date-based trigger? Not an easy one to answer in most cases and for most organizations.
Would love to hear your perspective on how folks are addressing this!
Cheers,
Joe
Joe & Pat,
This is very good discussion. We are getting started with IER implementation with FileNet as ECM for business content. We are going to leverage both fixed data driven retentions and event trigger retentions given the wide specturm of content we manage. For event-based retention & disposition, integration from the respective system of records will drive metadata/property of the content, and triggers of + x years will be set based on these metadata/property. We did a proof of concept, and are executing a full implementation.
Cheers, Bala
“how do they handle requirements like “death of the insured + 7″ or “termination of the policy + 3″? Are they letting FileNet know about the death of the insured or termination of the policy, or simply setting up a date-based rule to meet the spirit of the law, e.g., “policy origination date + 100″? “There are several different ways to handle these event triggers for example you can take a feed from an external system that feeds directly into P8 using the API or you do it via an xml file that is imported into P8. another item to be aware of is the level of aggregation, when you have triggers like “death of insured” or “termination of policy” your aggregation is going to be at the record level and not at the container level. as a result you’ll be running disposition sweep more frequently such as monthly or weekly. and you do auto-destroy since there is no way anyone can review and authorize the destruction of hundreds of thousands of records and the workflows can’t handle it either.
a former P8 systems architect
Reblogged this on 3mpecee Information Solutions and commented:
A fascinating read which I encourage all managers of records to read. Agile Ramblings critiques not only SharePoint as a Records Management Tool, but suggests that’s there isn’t one EDRMs/ ECM/ DRM package out there that Does Records Management 100% effectively.
I’d be extremely interested in your thought on how well you think your electronic record system does its job.
Thanks for reblogging and for the kind words–I appreciate it!
You absolutely can do real records management inside SharePoint. As we discussed with your firm on Friday, we have built the full set of capabilities inside SharePoint to enable the automated disposition of content based on an enterprise retention schedule. We have certified these add-in capabilities according to the DoD 5015.2 RM standards (Chapters 2 & 5) with Classified to be certified in May for our products. This includes extending record relationships in SharePoint for event-driven disposition, so that an ERP system event could be raised to initiate a disposition lifecycle in SharePoint. (What part of certification do you not believe in?) We have several clients that have acquired enterprise licenses and are currently implementing these products. We would be happy to demonstrate them to you at MER next month in Chicago or in a Lync meeting at the time of your choosing. SharePoint is a great platform for records management, and there are multiple companies that are implementing these solutions. Whether those companies use these capabilities to fully automate the deletion of records without human intervention is more of a question for their lawyers and records managers than the capabilities of SharePoint. SharePoint, as extended by add-in products, now does this as well as competing products. I think Bruce Miller basically agrees with this.
Mike,
I know you can do RM in SP with add ons–there are a number of them out there in addition to the Gimmal product you demoed for us last week–which is why my post specifically zeros in on out of the box SharePoint.
But even with an add on to SharePoint (or with RM via big ECM), the people/process problems remain. And you are right to point out that this is less a software capabilities problem than a organizational change problem, but it’s a problem nonetheless and it’s a real hurdle to actually doing automated records management. Having software in place that theoretically can automate records management without actually automating it (for people/process reasons) is as good as not having it, in my opinion.
Thanks for jumping in and sharing your perspective…feel free to drop a link to Gimmal’s RM for SharePoint product–I’m sure folks would like to see it.
Cheers,
Joe
Joe – per your request, here is the link to our products page.
http://www.gimmal.com/Products/Pages/default.aspx
Cheers, Mike Alsup
Hi Joe, thanks for posting on this subject, it’s a favorite in my (Dutch) blogs – from a recordsmanagement perspective – as well. I guess all Microsoft/SharePoint bashers and RM-vendors were trigger-happy retweeting the various sources on which your blog was posted. Until reading halfway about ‘the bigger problem’… and you are so right on that. I think Bruce Miller did a perfect job in summarizing and analysis on the missing recordsmanagement functionality in out of the box SharePoint. And you are right too on the point that it is not only a matter of functionality in this or other recordmanagement applications. It is also about making retention and disposition work in a digital environment. Reminds me of an organization I did recordsmanagement consultancy for on their SP2010 implementation for recordsmanagement. I asked the recordsmanager – managing paper records at that time – how he organized disposition.’I just read the label on the file, and put it in the bin. Or not.’ he answered.
Eric,
Thanks for your perspective and your kind words. I’m glad you stuck with the post despite it’s intentionally provocative opening to get to the heart of the matter, which is about the total context of records management regardless of system. And I totally agree with your take on the challenges of making retention and disposition work in the digital (as opposed to the paper) environment–not easy to do at all!
Cheers,
Joe
Great article Joe. I completely agree, and as we have discussed, these problems are the reason Collabware exists. If anyone is interested in seeing how you can achieve true records management success with SharePoint, check out http://www.collabware.com
Graham,
Thanks for chiming in here and providing a link to Collabware. I really liked the product overview you gave a the KnowledgeLake user group in Vegas last fall, so hopefully folks check you guys out for dong RM (and more) in SharePoint.
Cheers,
Joe
Thank you for the thoughtful article Joe… long time listener, first time caller. Please allow me to make a few comments.
First, we have seen considerable movement with non-Fortune 1000 firms toward enterprise wide, automated records management on SharePoint. There are many reasons for smaller enterprises to move faster including:
– less substantive legacy systems in place to start
– fewer staff committed justifying past decisions
– a 10,000 employee firm is generally able to move quicker than a 50,000 employee firm
– smaller enterprises sometimes are less regulated which simplifies compliance and policy
Moreover, if you move even further down market to enterprises with 100-5000 employees, SharePoint has been able to provide smaller enterprises with cost-effective access to full blown ECM/RM (and more) which previously wasn’t available due to the licensing and implementation costs for these legacy systems.
Secord, given that SharePoint 2010 was released about three years ago on April 16th, 2010 with the first service pack on June 28th, 2011, we should expect large enterprises who were early movers and went “whole hog” to the platform to just now be getting close to full roll out assuming that the recession hang-over didn’t cause them to defer portions of their implementations for a later date (e.g., disable the creation of new PSTs or file shares, but not deal with the existing).
Third, and finally, I would turn your title around to state, “If you can’t do records management in SharePoint, you can’t do records management.”
SharePoint is a platform that is already available to almost every organization, has somewhat high adoption, is visible to everyone, scales, and has a considerably lower total cost of ownership than competitors. If you can’t leverage a platform like that for records management, you’ll get nowhere with the other outdated, under-adopted legacy ECM solutions.
With respect to out-of-the-box (OOTB) SharePoint is a highly configurable and customizable platform. Much can be accomplished OOTB and only small modifications are required to account for event-based retention and disposition and other such considerations (we have done this work for a number of clients). As with all ECM/RM solutions, there is an implementation and services component to a successful SharePoint deployment. This component might look different with SharePoint vs. FileNet/OpenText/Documentum/etc, but there is such a phase to the project nonetheless. It could include configuration and customization with a system integrator, third-party products, or an in-house IT team cranking away. Regardless, SharePoint 2010/2013 is a vibrant, viable records management system with a considerably lower total cost of ownership and maintenance.
Bill,
Thanks for taking the time to craft such a thoughtful, detailed response and for sharing your perspective and experience with the group.
You’re 100% right that my perspective is skewed towards large organizations (my firm works 99.9% with Fortune 1000 clients), so I was glad to see you bring the view from the other side of the tracks.
As far as the meat of what you’re saying, I have only one bone to pick: “only small modifications are required to account for event-based retention and disposition and other such considerations.” My understanding, drawn, admittedly, from research and observation rather than from actually trying to build RM in SharePoint, is that substantial work would be required–so substantial, in fact, that folks like Gimmal, Harmon.ie, and Collabware have build product offerings around helping folks do it. So feel free to get down into the weeds if you have time and share some of your approach with the group so we can get a better idea of what you mean by “small modifications”–I know lots of folks would appreciate it.
Cheers,
Joe
Thanks Joe. Since the SharePoint 2010 release I have seen a number of SharePoint RIM implementations address automated retention and disposition without third party products and have been involved in a number with third party products. (And most haven’t gotten around to the file share issue you raised.) Both options are viable depending on the specific requirements and budget involved (neither are free), and from there we have the age-old buy-versus-build arguments available. I likely over stepped with “small modifications”, but event receivers (and/or workflow) in the context of a full SharePoint roll-out it should not eat up too much of the budget.
Of course, if full DoD5015.2 certs are required then buying is your only real option.
Also, I was not aware that Harmon.ie provided a records management solution on SharePoint or have I missed something? In a past life I worked at Colligo, and we always viewed Harmon.ie as an Outlook Add-in and mobile competitor.
Bill,
Check out this link for the Harmon.ie RM product: http://harmon.ie/solutions-record-management. Looks like they’ve been evolving into a more full-featured product line from their roots as an email plug-in.
Cheers,
Joe
Thanks Joe – The Harmon.ie for records management appears to be their client side application working with records center. Certainly a step in the right direction and adding some of the enterprise grade features that their competitors Colligo and Scinaptic have, but not a records management system in the Gimmal/RecordPoint/Collabware sense with advanced SharePoint (server-side) configuration/customization. Both Gimmal and RecordPoint (http://www.recordpoint.com.au/product-overview/partner-add-ins) leverage add-ins similar to Harmon.ie as part of their overall solution.
Speaking of long time listener, first time caller – GO BILL!!! Its nice to see our platform stewards step up to challenges like this!!!!
Great article! I don’t think SharePoint will ever be a full solution by itself for most organizations, but I don’t think that’s a bad thing. SharePoint 2013 still seems to follows Microsoft’s strategy as outlined in their 2006 ECM white paper “Enterprise Content Management – Breaking the Barriers to Broad User Adoption” which is to complete the common functionality that everyone can use and for the more specific compliance based requirements (which will differ for most organizations) to focus on an extensible platform for customization or third party products, this way you get exactly what you need and nothing more, which can minimize complexity and drive user adoption\ROI. The best third party app I’ve seen so far is Collabware CLM – huge fan!
Michael,
Good points about the strengths of software not doing it all–this has totally been Microsoft’s position with SharePoint…until recently, when they’ve started storming the ECM castle to try to do it all–and for global, Fortune 500 organizations at that. Unfortunately, the product just isn’t there yet, even with help from the partner ecosystem.
And thanks for sharing your thoughts on Collabware. Always good to have folks chime in with their product/vendor experience.
Cheers,
Joe
Thank you Joe, good one! Most of the RM solutions we worked on used only Sharepoint as interface to a professional backbone RM solution 🙂
Ulrich,
That’s a common way to architect a successful SharePoint/ERM solution…as well as just a strong ECM platform that leverages the best of SharePoint (front end, work in progress, dynamic document management, light workflow, etc.) and big ECM (records management, long term archival, BPM, etc.).
And thanks for the kind words–I appreciate it!
Cheers,
Joe
Great, thought provoking post.
Our organisation has been working in the document management / records management / ECM space for the past 9 years. We actually surveyed a range of organisations because we were so concerned about the lack of real retention and disposal being done in EDRMS. Sure enough, only one organisation claimed to be doing R & D on a regular basis.
We have built and sell a SharePoint application that delivers real enterprise-wide records management capabilities: iWorkplace Records Manager http://iworkplace.com/rdmanager.html (sorry, folks only available in New Zealand at the moment!). We use it ourselves and have many happy clients – a number of whom have been through recordkeeping audits successfully.
I think the question around why is retention and disposal done so rarely, if at all, regardless of system is a key question. My pick is that there are two main reasons:
1) when deploying ECM or EDRMS it is such a massive task to get the deployment over the line that implementing retention and disposal is left until ‘Phase 2’ at which point the money has run out and the appetite is gone…
2) a disconnect between the functionality of a system and the schedule. Often neither the vendor, nor the organisation, has any clear approach or understanding of how to go about implementing the schedule into the system. As an ex-records manager myself, we picked early on that this would be a barrier to the use of our iWorkplace Records Manager product. So we deliberately designed templates and workshop material to step people through how to understand their schedule and then how to render it in SharePoint with iWorkplace. It’s not rocket science but having a clear methodology really helps.
Sarah,
Thanks for joining the conversation and for your thoughtful and detailed response.
I love hearing success stories and about new products…even if us non-New Zeelanders can’t benefit yet! But I also appreciated your take on the challenges of the people/process side of ERM–spot on!
Glad you liked the post!
Cheers,
Joe
Good contribution by Sarah. To add, most organizations are still in “limbo” as to what to keep and what not to keep. In effect systems have become “dumping grounds” and the existence of the Records Management Module is even forgotten. I have actually found myself in Sarah’s point 1. We are now in phase 2 trying to utilize the Records Management Module. Cheers Richmond
Richmond,
And you’re not alone. Lots of folks I’ve worked with would likely cite “not knowing what should go where and for how long” as a bigger problem than the RM capabilities of their technology. As with so many other things, the technology capabilities are the easier part; it’s the people/process dimension that’s the real issue.
Anyway, thanks for jumping in and sharing your insight…and good luck moving to phase two and getting value out of your RM solution!
Cheers,
Joe
Excellent article. It may be provocative to some but not to anybody who has worked/consulted in this space. I would like to point out that the reason SharePoint fails is it is purchased as a “solution” when it is merely a tool and a rather weak one at that. Hoever, even with the most robust ECM system in place, it still takes people and processes to make them work and that is why so many ECM implementations, with or without SP, fall short. That said, SP should help organizations move in the right direction in creating practices and processes that will enable user deletion of a lot of redundant, outdated or trivial information (“ROT”). .
Cary,
Thanks for the kind words and for sharing your insight about SharePoint as a tool versus a solution…I think the same could be said for most ECM products out there! But that’sa subject for another post entirely…
Glad you found the post valuable!
Cheers,
Joe
Hi Joe,
My experience with SharePoint is much the same as yours, in that the only viable way to achieve compliant recordkeeping is through an add-on product. In Australia, there’s a product called RecordPoint which specifically addresses the compliance requirements of our Archives Act 1983 and the National Archives of Australia as an add-on to SharePoint.
However, I am curious what you mean by “automated disposal”. No objections to the idea of automated classification. However, records administrators always get a bit worried about anything that deletes records without a review process first. Do you literally mean “automated” or just a streamlined review, reporting and sign-off process?
Stephen,
I mean “automated” in the sense of “workflow” (i.e., kick off a system supported process that routes work to human actors and decision makers based on business rules) rather than “automatic” (i.e., take action based on business rules without human intervention).
Typically, for RM this looks like a message sent to someone responsible for the records eligible for disposal, whether a custodian, RM coordinator, RM liaison, a corporate records manager, someone in legal, etc., so that they can “okay” the disposal. Often the approval follows a chain of approvers to ensure that nothing slips through the cracks and leads to inadvertent spoliation. Once the approvals are in, “disposition” can be a multi-step process, often starting with “deleting” the record so that it can’t be accessed by anyone outside of RM or IT storage management, then followed by a purgatory period of 30, 60, 90, etc. days to ensure we have an undo in case we missed something, and then ending with real and permanent disposal (which leaves an audit trail, of course, to ensure that although the record is gone for good, we have all the relevant information about what it was and how and why we disposed of it when and how we did).
I hope this helps!
And thanks for sharing the info on RecordPoint…I know lots of folks will be interested to check it out!
Cheers,
Joe
Thanks Joe, I appreciate the clarification.
In which case, RecordPoint definitely meets your definition of an automated disposal process as well. It’s well worth checking out. I have a PDF on when RecordPoint is worth investigating for your organisation here – http://knowquestion.com.au/downloads#jswc
Stephen,
Thanks for the info on RecordPoint–I know folks will appreciate it!
Cheers,
Joe
Let’s not kid ourselves. True records management is a very complex practice. RMA is a process that is only addressed correctly when it is based on a flexible BPM platform, since that is what RMA really is… a lifecycle process from capture to disposition.
SharePoint is simply a common (comfortable?) front end for many organizations based on its longevity and Microsoft’s marketing machine. If you have attempted to develop in the SharePoint environment, and move every three years to the newest version (going into its fifth iteration), you understand its actual costs. So, acquiring a true, certified third-party RMA solution, and using the SharePoint Portal as the front end, is the long-term cost effective approach.
If you consider all of the record ingestion points, volumes (including e-mails), levels of security, e-discovery requirements, complex rules, reporting, event triggering by external sources, materialized data as a record, and many, many more complexities of a culture that has to adapt itself to a user perceived non-productive benefit, SharePoint is not the answer; it is only a point of entry.
That all being said, we believe we can be of help “doing records management in SharePoint” here at Feith Systems. Our Records Management solution, the BridgeLogiQ RMA iQ platform, has been under enhanced development for 27 years, and is certified at all three levels of the DoD 5015.2 specification. We feel, and we know you will feel, RMA iQ is the right platform for conquering the RMA limitations in SharePoint.
If you have more questions about Feith’s BridgeLogiQ RMA iQ solution, you can contact us directly, either through my LinkedIn profile, or by calling 215-646-8000 or emailing more-info@feith.com. We’re more than happy to give you an RMA iQ SharePoint demo, or point you in the direction of one of our educational webinars. (Thanks for bearing with the momentary product placement in the discussion.)
Sorry, it wouldn’t link our LinkedIn profile, but you can find more at http://www.feith.com (hey, if our friends from Gimmal can toss it to their website, so can we lol!)
Joe’s response to my note said, “Thanks for jumping in and sharing your perspective…feel free to drop a link to Gimmal’s RM for SharePoint product–I’m sure folks would like to see it.” So I did, as other vendors have. We appreciate the detail that you and others have added to the conversation.
Thanks for jumping in and sharing not only your thoughts about RM and SharePoint (which are spot on), but also information about your product. Beyond the “big three” ECM vendors, there’s lots of RM-focused solutions out there. And I know that folks are always eager to learn more about what’s available–no apologies needed for product placement.
Cheers,
Joe
“as it is said “99% of organizations are not doing automated records management on an enterprise scale. And this is true whether they have SharePoint, IBM FileNet/P8/CMOD, EMC Documentum, OpenText, Hyland OnBase” and the word “automated” in “automated records management” means a lot of different things for different people. I would rephrase the statement and transform it in a question “How can we take the most from SharePoint in order to automate records management””
This is a good point–sorry it’s taken me a while to respond…
By “automated”, I mean using business rules and an RM workflow to manage the disposition process, not records being purged without human intervention.
I also like framing the question in terms of how we can get the most out of SharePoint (or EMC, IBM, OpenText, Hyland, etc.)–but no matter how you phrase it, people/process issues are central to what you can/can’t do around your RM platform.
Thanks for jumping in!
Cheers,
Joe
Using Oracle’s Sharepoint adapter… yes. No sweat. 😉
Dwayne,
Do tell! I know folks would love to hear more about how you’re using it and what you’re using it to do in SharePoint.
And thanks for jumping in!
Cheers,
Joe
Dear Joe,
One of our colleague reblogged this on :
http://www.linkedin.com/groupItem?view=&srchtype=discussedNews&gid=2368275&item=230649782&type=member&trk=eml-anet_dig-b_pd-ttl-cn&ut=3GV8t3vW8RvlI1
It generated an interesting and animated discussion… as you’ll can see!
Germain,
Thanks for the heads up! I found the post (after joing the ARMA Montreal group) and appreciated the great discussion there!
Cheers,
Joe
Hi Joe
I enjoyed your blog entry and I am also enjoying the interest its generating (I have lost count of the number of times I have had the link sent to me)
I am the product development manager for HP’s offering in this space, HP TRIM. I often talk to customers and partners that think they can use SharePoint alone to perform records management and find myself explaining why this is not the right approach.
RM principles dictate fairly strict guidelines for how information should be organized and stored. This is designed to allow (amongst other things) the application of retention, classification, security and access controls.
SharePoint on the other hand provides the flexibility and constructs to allow users to organize and create information in the way that they want to (which is almost always not IAW RM principles).
Typically what I have seen is that when an attempt is made to use SharePoint for records management, the organization attempts to force RM constructs on SharePoint users by doing things like creating site hierarchies that reflect classification structures for example.
Instead of ending up with a system that allows leveraging the capabilities of SharePoint, they end up with a system that uses SharePoint only as a rudimentary web based records manager and they compromise the benefits that they turned to SharePoint for in the first place.
For the best of both worlds, RM should be the domain of a specialist RM platform (such as HP TRIM) with a connector that allows users to use SharePoint as they want to use it, not how a records manager tells them they should use it.
Until these two different sets of requirements are recognized and catered for, my thoughts are that SharePoint won’t be taken seriously as an RM platform.
Jaimie,
Great comments–so thanks for taking the time to craft this thoughtful response.
You’re dead on when you describe organizations deploying SharePoint with an RM-centric information architecture that renders it a rudimentary RM system at best, and one that compromises the benefits they got SharePoint for in the first place. I see lots of RM orgs considering this approach all the time.
You also get it right when you describe the ideal target state architecture, i.e., SharePoint for dynamic document management and a more robust platform for RM. I know that there are products out there that ostensibly make this dual architecture unnecessary by plugging in more advanced RM capabilities to SharePoint; but, for my money, I think orgs get more out of SharePoint plus ECM rather than with SharePoint plus plugins (and reduce risk, because they have only two vendors rather than a collection of them, of various sizes and stability).
And I’ll say the same thing I’ve said to other vendors here, if you want to pitch some of your success stories or capabilities, fire away. I know that the Autonomy situation has garnered a lot of the media attention, so folks may not be as aware of what TRIM is up to there days.
Cheers,
Joe
Interesting article and equally interesting feedback both here and on the linked-in discussion where I first picked this up. I agree with your perspective on SharePoint only RM implementations not cutting the mustard, but to be fair Microsoft have their hands slightly tied behind their backs. They have to build a product which provides RM capability but is generic enough for global use. In a previous life I helped put a single RM product through DOD5015.2, UK TNA2002, VERS and paid homage to MoReq (Autonomy RM – previously Meridio) so I understand the complexity of trying to engineer a single global solution.
Microsoft have gone for the middle ground (after failing to usably certify SharePoint 2007 against DOD5015.2) and provided what I would describe as the foundation platform (at least in the 2010 & 2013 versions) that give the building blocks for others to extend. They also very purposefully left the door open wide enough for partners to do so with their blessing.
We at Automated Intelligence (UK/Europe based) and others such as Gimmal (US) and RecordPoint (AUS) have done just that for our respective territories and their unique requirements. Albeit done right, a solution can bridge more than one continent…watch this space.
It’s very clear with the coming of age of SharePoint that the old concept of integrating a separate ECM product with SharePoint to meet RM requirements is just not viable or required any more. It’s neither functionally appropriate, technically advantageous or cost effective for a customer to do so. The only people that I’ve seen do this in the last few years have done so largely to justify the outrageous amount of money they have invested in their legacy platform of choice. It certainly not because the business or the users want it… as it happens the users are all very happy with the ease of use that SharePoint brings.
Customising SharePoint doesn’t solve the problem either, I see many customers who are locked into specific versions of customised SharePoint and are paying a hefty price to move forward to the latest versions.
Automated Intelligence provide a SharePoint extension framework for Records Management that does 2 things. Firstly it checks the boxes for formal records management in regard to record protection, retention, reporting and disposition. Secondly it interfaces at the user level to make records management something that the user can engage in without even realising it (having worked for multiple RM vendors over the past 15 years burdening the user is one of the biggest opportunities for failure in implementing successful RM).
AI.COMPLIANCE EXTENDER (http://www.automated-intelligence.com/ai-compliance-extender/) is currently being used widely across central government, local government and other bodies to provide this level of control and access.
Another key requirement is bridging the gap between SharePoint and Outlook (and increasingly SharePoint and mobile) is also key to ensuring successful compliance and user engagement to meet RM requirements. While we provide our own solutions for this, AI.SYNCPOINT for Outlook and AI.MOBILE, there are a myriad of solutions available as has been pointed out earlier in the thread.
One additional comment on the original article I would also like to address is the management of file shares, or lack thereof! For this very purpose we built AI.FILEMANAGER that allows our customers to apply ‘fully automated’ retention and disposition policies to file shares (and 40+ other ECM systems). It’s a huge gap in most organisations record strategy and strangely one that very few vendors have sought to fill?
Again this solution is today being used by large government agencies to provide compliance to their least controlled (from an RM perspective) data… So it is possible and it’s starting to happen (slowly, but it’s better than what’s gone before!).
AI.FILEMANAGER (http://www.automated-intelligence.com/ai-filemanager/)
Thanks again for the blog post and the opportunity to feedback.
Best regards,
Simon.
In the other thread of this string, I responded that Gimmal believes that Gimmal, Collabware, and RecordPoint are all advancing the art of managing physical and electronic records inside SharePoint. This is also true of Simon Cole and the team at Automated Intelligence in Europe. This is becoming a very capable ecosystem of providers solving enterprise-scalable records management challenges inside SharePoint, so that a third party repository of record is no longer required.
Simon,
First, thanks for not only taking the time to jump in and join the conversation, but for your thoughtful and extensive comments—I really appreciate it!
Second, thanks for the info on Automated Intelligence and the products you all offer. I appreciate the chance to learn more about what you all do and I know folks out there do, too.
Some thoughts about some of your thoughts…
A lot’s been said here on all sides about using third-party tools to extend SharePoint and do RM, so I won’t speak to that issue again other than to say that I agree that there are tools to help SharePoint do RM, but that the larger issues preventing folks from succeeding are mostly people/process (and therefore much more difficult to overcome).
Beyond that, your take on SharePoint – “big” ECM coexistence is an interesting one—one that I’ll speak to here, but want to handle in a separate post that hopefully gets another good conversation going…so stay tuned.
First, where we agree: I agree that it would be best not to have two systems for ECM. I also agree that in many cases, SharePoint would be a great sole ECM platform.
But I disagree that there are no benefits to having SharePoint plus “big” ECM (whether you integrate, loosely couple, or run them side by side). For things like ERP integration, transactional workflow, high-volume image management (tens/hundreds of millions of files), or vertical LOB solutions (like engineering drawing management), SharePoint by itself doesn’t (yet) cut it.
In response to this, folks may rightly chime in that there are partners out there that can extend SharePoint to do these things, so you don’t need “big” ECM; but leveraging the range of partners needed to meet these example use cases (all of which would be common at many F1000 organizations) introduces a lot of risk into the equation, because these partners will tend to be smaller, less stable companies than say, OpenText or IBM. For my money, I’d rather rely on Microsoft and OpenText than Microsoft and Partner A, B, and C (and D, and E, etc.). Which of these partners will be around in 3 years, 5 years, 10 years? Which will be acquired? Go out of business? Change product direction? This is not a bet I would take if I were a global FiServ, Pharma, Energy, Manufacturing, etc. firm.
So I don’t think the main reason organizations go this route is because they’re throwing good money after bad. I think the complexities of what they’re doing in “big” ECM is difficult, expensive, or impossible to replace in SharePoint currently. Things may be different in 5 years, but I think that’s the reality of where we are today with SharePoint for the large enterprise.
I also disagree that business users are universally (or even mostly) happy with SharePoint. I’d say it’s been 50/50 at organizations I run across—which has less to do with SharePoint than with how it’s been deployed and supported. But in that respect, it’s no different than any other piece of technology. Some love it, some hate it, some tolerate it…and these have to do with more than just the technology in question.
Thanks again for joining in the conversation!
Cheers,
Joe
Joe said, “For things like ERP integration, transactional workflow, high-volume image management (tens/hundreds of millions of files), or vertical LOB solutions (like engineering drawing management), SharePoint by itself doesn’t (yet) cut it.”
At a huge Foods company in your hometown, we (and our partners) replaced their FileNet system connected to SAP with a SharePoint solution connected to SAP. No change to the SAP user experience. They process tens of millions of invoices and related documents every year. They use workflow on both the SharePoint and SAP sides. SAP itself spreads the content across volumes, so they could easily scale to over 1000 servers. What part of the enterprise scalability of SharePoint in this context are you still questioning?
Mike,
Quick question: what are they using to integrate SharePoint with SAP?
Joe
Joe –
Gimmal provides connectors that link transactions in SAP to documents in SharePoint via the SAP ArchiveLink connectors. We also enable most SAP business objects to be exposed as .NET objects in SharePoint, and enable SharePoint search and workflow to be performed on these objects based on synchronizing the metadata between the two systems. We have integrated this with our SharePoint content governance. More information is here:
http://www.gimmal.com/Products/Pages/default.aspx
Thanks,
Mike Alsup
Mike,
Thanks for the further info on how they’re integrating SharePoint and SAP.
My point in the post is not whether SharePoint can do transactional imaging, heavyduty workflow, etc., with a partner or partners, but whether F1000 on the whole would be willing to live with the higher vendor and product risk of augmenting SharePoint with Microsoft partners to do these things. My take is that most F1000 firms will choose to go with “the big guys” (i.e., IBM, OpenText, EMC, Oracle, HP, etc.) rather than smaller partners.
My other point is that I don’t believe Microsoft cares too much about displacing big ECM anymore. I think they are much more focused on cloud CM providers like Box and Dropbox. Would you rather kick out CMOD and get the 10M images in it or ingest 100TB shared drive? From a revenue stream perspective, I’d take the latter any day.
Cheers,
Joe
Joe –
The reason that so many companies are going the SharePoint route over the legacy ECM route is that the cost to replace Open Text or FileNet with SharePoint in an SAP environment is typically 3-6 months of software maintenance (depending on the usual scale and complexity factors). Zero change to the user experience. Unload from Open Text into SAP, reload to SharePoint. Support for a million scanned images per month is no problem. We are currently converting 30K seats from a leading ECM Suite to SharePoint in Houston because they are tired of that $3MM per year it cost them.
As for Microsoft being focused on Box instead of F1000, I received several emails from the Microsoft field today about comparison with leading ECM Suites in some specific use cases. Microsoft is completely opposed to solutions that take content out of SharePoint into a 3rd party repository for almost any reason. Box is an afterthought in a competitive and minimally profitable market.
Thanks, Mike Alsup
Mike,
A clarification: I wasn’t saying that Microsoft wants to partner with Box, Dropbox, etc., but that they are working hard to put them out of business and eat their lunch and that I think Microsoft is much more focused on squashing these cloud CM folks than “big” ECM these days.
And it makes sense when you consider the fact that the lion’s share of unstructured content at most F1000 firms is shared drives/c drives and is also the fastest growing repository type (except maybe email). So if I’m Microsoft, I’d much rather have a company shuffle all their shared drive and c drive content into O365 (and have it grow at 50% a year) than replace an ECM system. There’s not only going to be more money there but a consistent and increasing revenue stream for the forseeable future.
Cheers,
Joe
Thanks for finally talking about >You can’t do records management in SharePoint | agile ramblings <Liked it!
Thanks for the kind words and support–glad you found the post helpful!
Kudos to everyone who participated in this discussion, it really turned into a very valuable post of information. Especially for all managers of records. I have saved it and will share with readers on my sharepoint blog. Keep up the great work!
John.
John,
Thanks for the kind words and for sharing with your readers…I appreciate it!
Cheers,
Joe
I would be surprised if anyone would define records management as “actual, system-enabled automated disposition according to your retention schedule. ”
Is records management simply about managing records until they are disposed? Records management is first and foremost about supporting the business for which it is created. We may wish to have records management as the queen of the business management systems kingdom, but it is not.
Records management serves the business purpose. Is sharepoint designed for that purpose or is it only one purpose among many. To put it even more provocatively, does any electronic system do records management? No. Why? Records management is done by people not by machines. Machines or computer programmes only assist what humans want to achieve. People can and do use Sharepoint to manage their records. How well they do it, is *not* a function fo Sharepoint.
Sharepoint will not save you from a poorly designed records management system nor will it save you from a corporate culture that is uninterested in records management. One only need to look at the collapse of Lehman Brothers and understand that their corporate culture doomed them rather than records management. The quality of their records management was a function of their corporate culture not the other way around.
As some explained to me, all that Sharepoint will do is get you to where you were already going before you had it and probably faster.
Lawrence,
Thanks for jumping in with your thoughtful reply. You raise a number of issues, and I want to address each of them (hopefully) as thoughtfully as you do.
First, you raise an interesting point:
“I would be surprised if anyone would define records management as “actual, system-enabled automated disposition according to your retention schedule. ”
Is records management simply about managing records until they are disposed? Records management is first and foremost about supporting the business for which it is created. We may wish to have records management as the queen of the business management systems kingdom, but it is not.”
I’m not sure I understand what RM would be if not managing records until they are disposed, although I would put it differently, i.e., managing records throughout their lifecycle, from creation to disposition. And if, as an RM function, you are not managing the disposition of records at the end of their lifecycle, you are not supporting the business, because you are exposing the business to a great deal of risk. Legal–because you’ve failed to keep the volumes of likely discoverable information in check–and operational–because users have trouble finding what they need from among the swelling volumes of over-retained records and because IT is increasingly unable to maintain its infrastructure given the volume of content on it.
So, for me, if RM isn’t about managing records from birth to death, what is it about? And how exactly does it support the business?
Second, I agree that systems only support what humans design them for. But if a system is poorly designed, then what it can contribute towards human designed ends is limited. And in some cases, the imitations are so extreme, that no portion of the human ends can be accomplished. This is my point about SharePoint and RM: yes in order to so ERM we need a human vision and oversight; but if the tool we pick is simply incapable of adequately supporting a sound RM vision, we will get nowhere.
Finally, I also agree that SharePoint or any other system will not save you from a poorly designed RM program. But my point was the opposite: folks with sound RM programs are being sold SharePoint as a tool to enable their sound RM program, and that’s just not true. It is one part of the equation, but it requires either add-ons, custom coding, or a more robust repository (aka, “big” ECM) to succeed.
And if any of this response is unclear or you want to continue the conversation, please jump back in–I love discussing this stuff!
Cheers,
Joe
Joe,
Thanks. First, if you had mentioned life cycle I would not have posted. Too often people seen RM as about managing to destruction rather than using the documents. ie RM is archives.
Second, sharepoint may be poorly designed. My guess is that is does somethings well and other things not as well. However, that is less a problem with software than it is with a business model. Sharepoint works for what it intends, sharing or accessing documents.
Third, folks are buying the system. If they make poor choices, who is at fault? Most RMers I know would ask a lot of hard questions about sharepoint as a RM system. Lots of doubts in the community.
Fourth, as an aside, too much is made of RM and confused with IM or DM. In reality, most senior managers want a system that will do all three. Alas, no such system exists except the corporation in general and even that is doubtful.
Perhaps you could suggest the system that does work. I am still looking.
You may be interested in this article on the issue.http://thoughtmanagement.org/2012/04/21/here-is-the-next-revolution-in-records-and-records-management/
Best,
Lawrence
Our company just got into using sharepoint. I just want to thank this community and the publisher on this site for sharing all of this sharepoint tips and information. I have learned so much just from the discussion here.
John.
John,
Thanks so much for your kind words–glad you found the post and ensuing discussion valuable…and good luck with SharePoint!
Cheers,
Joe
John,
I have an active SP 2010 site today and we are considering turning on Records Center. Need to know if that is an option for SharePoint 2010 STANDARD edition or do I need to have Enterprise edition?
My understanding is that the records center is available in SP 2010 standard edition.
IMO, the problem is not software related. Most company employees are not trained in using a paper-based file plan for official records even, so no matter what software solution you implement, it will never fix the problem.
Kristopeher,
I couldn’t agree more–we throw technology at the ECM/RM problem and expect it to fix everything (and blame it when it doesn’t), but technology is at best 1/3 of the problem. People/process context is the real indicator of success. Get that right, and the technology is easy.
Thanks for jumping in!
Cheers,
Joe