What is Bitcoin?

So you may have heard about Bitcoin, some of you might be quite familiar with it - but for those who aren't, it is a digital currency - a form of electronic cash which can be used to make online payments, transfer money to friends - in theory, anything that you can do with traditional money, you can equally do with Bitcoin.

Bitcoin uses many of the ideas initially developed for peer-to-peer file sharing networks to create a decentralized system, ie: one that is not controlled and cannot be manipulated by any person, organization or government. The software it runs on is open source and can be viewed by anyone, and so in this sense, it is very much considered to be in the public domain, both in terms of the network itself, and the software running on it.

I hope that is an ok explanation for the purposes of this post - but for those who'd like to find out more, there are some links below to some good sources who can explain it much better and in a bit more detail than I can:

• Bitcoin Simplified - http://bitcoinsimplified.org
• Bitcoin Wiki - https://en.bitcoin.it
• Bitcoin 101 - https://www.youtube.com/channel/UCgo7FCCPuylVk4luP3JAgVw

So What is BitPay Then?

BitPay is a payment processing service for Bitcoin payments - they are kind of aiming to be the Paypal of the Bitcoin world. Whilst Bitcoin is a public network, and you can theoretically broadcast transactions to the network yourself without having to rely on a third-party to do that for you (we are working on a payment processor for that too, but more on that coming soon), it requires a lot of setup, your own server, a fair bit of background knowledge - it's probably not for most people.

So BitPay aims to simplify the process of accepting Bitcoin payments by providing a gateway interface similar to any other payment processor. It's worth noting there are plenty of alternative payment processing providers for Bitcoin payments out there, but BitPay is one of the most popular, and it's the one we've decided to implement for now.

Installation

You can currently get hold of the extension from the Github url below - although we'll make it available through the public extensions directory once it's been better tested and is considered stable:

https://github.com/circleinteractive/uk.co.circleinteractive.payment.bitcoin

Place that in your extensions directory (either by cloning through git, or by downloading the zip file), and then enable the extension. This will make two new payment processors available - 'Bitcoin' and 'BitPay'. Ignore the 'Bitcoin' one for now.

Also, be sure to configure your extension resource URL (will try to add a warning for that in due course), as the extension will not work correctly if this is not done.

To add the BitPay processor, go to Administer -> System Settings -> Payment Processors and click 'Add New Payment Processor', then select 'BitPay' on the next page.

You should see a screen like the one below:

All you should need to do is name your processor, enter an optional description, and enter your BitPay api key. An API key can be generated through your account dashboard on the BitPay site.

You may also see the following warning:

See the section below marked 'Payment Notifications' for more info on this, but in short, we have designed the processor to work with or without SSL, so if you get this warning, you should make sure cron is running, and configure it to run as often as you think sensible - we would recommend at least every 15 minutes, but preferably every 5-10 minutes to ensure payments are completed as promptly as possible after confirmation is received.

Overview

So here is a test event page with the BitPay processor enabled, along with some others:

You'll see it converts the prices to BTC at the current exchange rate when you have BitPay selected as the processor. So you set the prices in your local currency, and the price in BTC is calculated at the time of signing up. And that is also displayed on the confirmation screen before checking out, here:

Payment

To make payment, the user will be redirected to the following page on the site:

You can make payment either by using a mobile wallet app to scan the QR code, clicking the 'Pay with Bitcoin' button - this will open up a wallet app on your computer which is configured to handle Bitcoin links, with the amount prefilled - or you can manually send the amount on the invoice to the address at the bottom.

For now, we're going to use a mobile wallet app to demonstrate - so scanning the QR code with your app prefills the receiving address and amount, like so:

Then all that is required is to click the 'Send Payment' button. When you've done that, the on-screen invoice should display a status of paid:

.. and then redirect to the usual 'Thankyou' page which we're all familiar with.

Confirmations

There will be a delay before the transaction is fully completed, as at the moment, the processor uses the default behaviour of requiring 6 network confirmations before considering the transaction as completed, at which point the contribution will be marked as complete in Civi, and confirmations emails sent (if configured).

This is to guard against fraud and something known as 'double spending', and is quite important when exchanging Bitcoin for physical goods, as you don't want to be in a situation where you've sent out the goods, then later discover the transaction to be fraudulent.

However, with the type of transactions Civi handles (contributions, event registrations, memberships), it is not so important, as with contributions, you don't generally get anything in return - and with memberships and event registrations, these are things that can be revoked or cancelled if the confirmation process subsequently fails.

So we will probably introduce an additional setting for this on the payment processor configuration page, whereby you can choose to complete the transaction immediately, as soon as it is marked as paid - however, you should be aware of the potential dangers of doing this.

In short though, there is currently a delay of 30 mins to 1 hour while confirmations are received, before the transaction is marked as complete in Civi.

Payment Notifications

For security reasons, BitPay will only send payment notifications to an SSL (https) url - this is when the BitPay site contacts your site to provide updates on the status of the invoice - basically, whether the payment has gone through or has failed.

So if your site has an SSL certificate installed, you shouldn't need to read any further - everything should just work as if by magic. Just be sure to check that anonymous users have the 'Profile listings and forms' permission, which I think is the case for all processors that implement payment notifications.

However, we're aware that the majority of CiviCRM sites don't have SSL certificates, and thought this might be a bit of a drawback for a lot of users / potentially limit the audience somewhat, so have designed in a fallback system for those cases.

There is a Scheduled Task (cron job) which should enable itself automatically if it detects you don't have an SSL certificate - all you need to do is configure cron to run as often as you can. Be careful with this, as it does have a potential impact on the performance of your site - but as stated above, at least every 15 minutes would be good, every 5-10 minutes would be better, but it's up to you to decide what's best for your use case - the less often it runs, the longer payments will take to be marked as complete in Civi after BitPay has received the necessary confirmations.

That is all

I hope the extension proves useful - please try it out, let me know about any bugs or problems. There is a forum thread here for the moment where you can do that, although I will probably start a new one just for this extension if it proves popular / if there are a lot of issues.

Also, many thanks to Circle Interactive, who let me spend a fair bit of work time doing this. It wasn't written for any particular client - we just did it because we wanted to, and thought it might be a nice addition to Civi.

I have a few payment processing related projects on the go but having just gotten the first one to alpha stage I thought it would be a good time to share some thoughts (& try to get them straight in my mind). The project I am referring to is developing a Cybersource Secure Acceptance POST payment processor. For this payment processor I had a big challenge and also a big opportunity.

The challenge

The Cybersource Secure Acceptance POST gateway is one of an increasingly common breed known as transactional redirect or sometimes direct post method. The idea is that you gather most data on your site and then present the user with a credit card form on your site which POSTs to the Cybersource site. This way the credit card form is presented on your site and the customer never knowingly (depending on internet speed) leaves your site but none of the credit card information ever hits your server in any way. This gives you PCI compliance without the ugly redirect (& risk of the offsite form being fugly). The only problem is that CiviCRM isn't built for it.

The opportunity

In the not so distant past I discovered an exciting library of payment processor gateways https://github.com/omnipay/omnipay - omnipay written by

Adrian Macneil

. Omnipay normalises your interactions with payment processors so that you need less code to integrate them and you can leverage contributions from an open source community that extends far beyond CiviCRM. In this case there was not an existing Cybersource gateway but someone called Rafael Diaz-Tushman had a shell of a repo which I submitted code to and now he is actively working on too. So, already we can see this has enable collaboration outside our community.

What works?

The 2 processors I tested are Paypal and Cybersource - although I tried a few others like BitPay, Authorize.net and Payment Express. Bitpay worked locally but not on a second site and I didn't figure out the problem with that or the others. Generally I didn't have the credentials / time to explore non-relevant processors too much. However, Paypal & Cybersource both have very different flows and use the same code. If you check the processors list there are about 35 processors there. In some cases they have endpoints for functionality like recurring and refunds that we haven't fully developed in CiviCRM yet.

https://github.com/eileenmcnaughton/nz.co.fuzion.omnipaymultiprocessor

Learnings

The first thing I discovered working with Omnipay is that the maintainer has decided that thinking about processors in terms of off-site processors and on-site processors is not useful.

• Off-site gateways such as PayPal Express, where the customer is redirected to a third party site to enter payment details
• On-site (merchant-hosted) gateways such as PayPal Pro, where the customer enters their credit card details on your site

This is quite a different approach to that adopted by CiviCRM which has very different flows for on-site and off-site processors. However, I quickly came to the conclusion that the different flows are not actually that helpful and only capture a small subset of payment processing possibilities. In fact the doTransferCheckout in my implementation of the OmniPay library looks like this:

functiondoTransferCheckout(&$params,$component='contribute'){  $this->doDirectPayment($params,$component);  thrownewCRM_Core_Exception('Payment redirect failed');

}

The 2 key differences in how Core processes Onsite and Offsite payments are

1) Form Building differences

2) Pre-handover processing differences

Form Building differences

CiviCRM says 'if you are an onsite processor you need to have address fields + credit or debit card fields and I will create a billing address for you & for your contribution otherwise you don't need these fields'. As it turns out the form fields requirements of payment processors vary quite a lot and in the case of Cybersource - which in the end I had to get CiviCRM to 'think of' as an offsite processor you still need specific address fields (which are then signed). Conversely we get fairly frequent requests from people wanting us to remove billing fields as their processor doesn't require them and they prefer less fields. Basically core needs to 'ask' the processor which fields to display rather than decide based on the processor's 'type' combined with it's payment type. I achieved some of this within the extension (in particular replacing BillingBlock.tpl with a block that has no hard-coded fields in it & which displays fields based on variables assigned to the template rather than template logic. However, I feel like I'm only starting to think through the metadata involved in 'instructing the form layer' and this is something I'm going to write more about - probably in a second blog.

Pre hand-over processing differences

If you love a contribution set it free - if it comes back, change it to completed. If it doesn't - hey who knows maybe it's gone off-site & there will be an IPN later.

The above is my revised version of how CiviCRM should interact with the payment processor. In other (rather less opaque) words CiviCRM should build the pending contribution & related assets prior to handing over to the payment processor regardless of what 'type' it is and then update to completed based on 'what happens next'. How does this compare to current action? Well - at the moment CiviCRM creates a pending contribution if the processor is 'of type notifiy' but if it is of type 'form' it doesn't create one unless it is confirmed by the doDirectPayment function. This has a couple of effects. Firstly it makes the code more complex because the form layer is iffing & thenning all over the place. Secondly it makes it hard to create payment processors that don't fit standard flows.

What are the downsides of this flow change? Well, firstly this change shouldn't hurt existing processors (which is our main block to making some other changes I'm mulling) but there are 2 potential gotcha areas

1) Probably this would leave more failed transactions in the database and arguably this is a bad thing. At the moment failed credit card transactions are either not stored or rolled back out of the database - I've only been on the 'why do we lose our audit trail' side of this but there are others who prefer it this way. My suggestion would be to have a cron that cleans up failed transactions rather than not record them or introduce another setting as to whether they are desirable

2) Double transactions. Some processors create the option of a separate transaction for memberships vs contributions. I can't see this being able to be decoupled from payment processor type 'category' in the short term. In the longer term I think we could think about metadata & capabilities. I'm not going to go further into metadata & capabilities this blog but I do want to publish a blog on it

What does Omnipay Do

Basically it normalises your data & the interactions with the gateways. It doesn't matter what your gateway calls the First Name field - you set the same fieldname everytime and Omnipay takes care of that for you. So, here is my doDirectPayment function - that handles onsite processor Paypal, offsite processor Bitpay & transactional Redirect Processor Cybersource.

function doDirectPayment(&$params, $component = 'contribute') {$this->_component = strtolower($component);$this->gateway = Omnipay::create(str_replace('omnipay_', '', $this->_paymentProcessor['payment_processor_type']));$this->setProcessorFields();$this->setTransactionID(CRM_Utils_Array::value('contributionID', $params));$this->storeReturnUrls($params['qfKey']);$this->saveBillingAddressIfRequired($params);try {$response = $this->gateway->purchase($this->getCreditCardOptions($params, $component))->send();if ($response->isSuccessful()) {// mark order as complete$params['trxn_id'] = $response->getTransactionReference();//gross_amount ? fee_amount?return $params;    }elseif ($response->isRedirect()) {if ($response->isTransparentRedirect() || !empty($this->gateway->transparentRedirect)) {        CRM_Utils_System::redirect(CRM_Utils_System::url('civicrm/payment/details', $response->getRedirectData() + array('payment_processor_id'=> $this->_paymentProcessor['id'],'post_submit_url'=> $response->getRedirectURL(),          )));      }$response->redirect();    }else {//@todo - is $response->getCode supported by some / many processors?return $this->handleError('alert','failed processor transaction '. $this->_paymentProcessor['payment_processor_type'], (array) $response, 9001, $response->getMessage());    }  } catch (\Exception $e) {// internal error, log exception and display a generic message to the customer    //@todo - looks like invalid credit card numbers are winding up here too - we could handle separately by capturing that exception type - what is good fraud practice?return $this->handleError('error', 'unknown processor error '. $this->_paymentProcessor['payment_processor_type'], array($e->getCode() => $e->getMessage()), $e->getCode(), 'Sorry, there was an error processing your payment. Please try again later.');  }}

Omnipay features

Omnipay gateways support different methods and it is possible to query the gateways to find out what methods they support. The most relevant methods are Purchase, Create Token, Refund and Complete Purchase. Purchase is obvious. Complete Purchase actually means 'handle IPN'. Once again I was able to use a very generic function - although there is some argy bargy with the $_REQUEST which was because it's possible to run this from the api. (I should check if that's required)

public function processPaymentNotification($params) {$this->gateway = Omnipay::create(str_replace('omnipay_', '', $this->_paymentProcessor['name']));$this->setProcessorFields();$originalRequest = $_REQUEST;$_REQUEST = $params;$response = $this->gateway->completePurchase($params)->send();if ($response->getTransactionReference()) {$this->setTransactionID($response->getTransactionReference());  }if ($response->isSuccessful()) {try {      civicrm_api3('contribution', 'completetransaction', array('id'=> $this->transaction_id));    }catch (CiviCRM_API3_Exception $e) {if (!stristr($e->getMessage(), 'Contribution already completed')) {$this->handleError('error', $this->transaction_id  . $e->getMessage(), 'ipn_completion', 9000, 'An error may have occurred. Please check your receipt is correct');      }    }  }elseif ($this->transaction_id) {    civicrm_api3('contribution', 'create', array('id'=> $this->transaction_id, 'contribution_status_id'=> 'Failed'));  }$_REQUEST = $originalRequest;  CRM_Utils_System::redirect($this->getStoredUrl('success'));}

Refund and Create Token

These are the 2 areas I think offer most potential for development focus at the moment. At the moment we can't call an api to refund a payment & have that passed to the payment processor. With Omnipay the most difficult part of creating that api and having it work on multiple processors will be agreeing what to call it (anyone who has been party to API discussions will know that discussion is a forte).

Create token is the only option Omnipay offers for recurring payments. From the author

My feeling is that we are seeing most growth in the area of tokens at the moment and a great focus would be to build an Omnipay cron for processing recurring contributions (based on tokens stored during checkout)

Omnipay forever?

The Omnipay library is well-written and well tested. It uses modern coding standards and continuous integration.  It's being built into Drupal 8 and there is Wordpress integration along with a fleet of others such as Silverstripe.The only other library I have found which seems compelling is Payum. Payum has a few more gateways, and some more features around logging and they have integrated Omnipay into their own code. It also looks well written. I didn't perceive a big feature difference between the two - not features that seem needed. So, to my mind it comes down largely to maintenance. Omnipay has greater penetration but seems to have a single maintainer with relatively simple needs "My main use case is a shopping cart, so most of the existing gateways have been built at a lowest common denominator level of interoperability" (https://groups.google.com/forum/#!topic/omnipay/lcgkiKg7rt4) whereas Payum has less penetration but they seem to be building a business around developing Payum (it IS open source) so probably have more resources committed to it. So, the risk is that the Omnipay maintainer could get overwhelmed by what seems to be a pretty steady trickle of support requests. 

I'm trying to be even handed here but I've now spent a lot of time getting used to Omnipay and I like it and feel like it's a case of 'convince me why not Omnipay'.

The areas that I still need to work through is the aforementioned metadata because we are dealing with a situation where we are integrating multiple gateways and need to know things like 'which fields to I show for this processor', 'will this processor support 2 payments in one transaction'? and in the case of transparent redirect fields it needs to know things like whether to show a single field for expiry date or 2 fields. Some metadata options are features of the processor and others are features of your account (currency for example could be either). Another variant is Stripe which requires a javascript inclusion (and probably specific classes or similar).

Some other libraries that I didn't think were as promising

How do I add a processor (that has a Omnipay gateway) to Omnipay Multiprocessor extension

Basically add your processor to the .mgd file & see if it works ....

Note that I am using payment_type = 3 to denote that billing address fields should be displayed (but not credit card fields). The url fields are not used and the label fields need to reflect the gateway implementation - ie. the password label is 'Access Key' - which will translate to $gateway->setAccessKey('blah'); The name field is also important as it is omnipay_GatewayName

array('name'=> 'OmniPay - Cybersource','entity'=> 'payment_processor_type','params'=>array('version'=> 3,'title'=> 'OmniPay - Cybersource','name'=> 'omnipay_Cybersource','description'=> 'Omnipay Cybersource Payment Processor','user_name_label'=> 'Profile ID','password_label'=> 'Access Key','signature_label'=> 'Secret Key','class_name'=> 'Payment_OmnipayMultiProcessor','url_site_default'=> 'https://testsecureacceptance.cybersource.com/silent/pay','url_api_default'=> 'https://testsecureacceptance.cybersource.com/silent/pay','billing_mode'=> 4,'payment_type'=> 3,    ),),

If you are one of the early adopters of CiviVolunteer on 4.4.x and planning to upgrade to 4.5, your help is needed to test out the 4.5 release of CiviVolunteer 1.3.
This release is to ensure that people are not held back from upgrading to the much anticipated 4.5 release. However the brand new release of CiviVolunteer 1.4 is right around the corner and will have significant new features. This release has been sponsored by the Manhattan Neighborhood Network, a community cable access station, http://www.mnn.org/ , and will be incorporated into the Community Media Drupal Distribution which MNN is a co-steward of. http://cmdrupal.org/

If you run a volunteer program or simply participate in one, please head over to the CiviVolunteer Roadmap [http://is.gd/civivol] and give us your 2 cents on which features should be in the next phase of development. We are taking feedback via the forums and on the developer wiki. The list of unfunded enhancements is already a healthy length but we need your help to prioritize it! Let us know how you would use them in your program, or what combination of features are essential together.

We are adding more flexibility and convenience to public volunteer forms, making it easier to collect information about your volunteers, and adding a little polish here and there to the already slick scheduling interface.

A good volunteer program gives feedback and recognition to volunteers. So we are building a "commendation" system to allow managers to easily recognize exceptional volunteers. Different widgets will be available to display these commendations on Volunteer's profiles (requires CMS integration).

At the heart of CiviVolunteer is the scheduling interface. Designed for ease of use at the begining, we are now working on scaleability by adding filtering. It will now be easier to find contacts who are eligible for a selected role. You will also be able to search for volunteers without leaving the scheduling interface and apply filters for your search by qualifications or availability.

There is a newly updated version of the native extension called "Fancy Tokens". This new version (2.1) includes enhancements suggested by the community. Specifically the request from Xavier to include tokens for individual event registration pages, where the event ID can be easily changed. 

Have you ever needed to send an email from CiviCRM that includes a list of upcoming events? Then you know how much fun the copy/paste effort this entails, especially if you want to use a checksum in the links to register. Plus you know you get to repeat the whole exciting process for the next newsletter the following week or month. Or if you have someone else you need to train on this task, try explaining to a non-technical person how to hand-edit the URL query string to include the checksum paramaters. (is the contact ID query string param name "cid=x" or "id=x", well that depends....)          And even if you do not deal with events, you can have the same fun teaching someone else to properly use checksums when you need to include a hyperlink to a contribution page or a stand-alone profile.  

If you want a much easier process for listing upcoming events with links to register, including links to contribution pages and stand-alone profiles, then try out the new extension called "Fancy Tokens" https://civicrm.org/extensions/fancy-tokens.    That page also includes more details about how to use the new tokens. I have tested this with versions 4.3 and 4.4. Since this is a native CiviCRM extension, it works with WordPress, Drupal and Joomla.

Never used checksums before? Then you are missing out on a great (but previously hard to use) feature of CiviCRM. Checksums give each recipient a unique URL that will be preloaded with the recipient's contact data (such as email, first name, last name or whatever data you are collecting.) The use of checksums saves time for the recipient when filling in your contribution pages, event registrations, and stand-alone profiles.   It may also reduce the chance of duplicate contacts in your database due to poor data entry habits of your recipients. 

Technical Note:  I had to include the .tpl file "/CRM/Mailing/Form/InsertTokens.tpl" within this extension  because the core version of this file has a hard-coded width for the token dialog window that is too narrow to display longer token names.  If this file can be eliminated, then this extension would work with more versions of CiviCRM.  

-Sarah Gladstone

http://pogstone.com

Last year we worked together with MAF Norway on the TriggerAction extension to help them with their fundraising donor journeys. They were able to find some initial funding so we developed like mad for a week to be able to start. The TriggerAction extension was a result of that hard work, and MAF Norway presented this at the London CiviCon on 25th September 2014.

Even though we are proud of what we managed to achieve in such a small timeframe and the MAF Norway is actually using the extension, we all think we need a second development sprint to improve the concept. We hope the session at CiviCon will help to find some additional funding and are really happy that MAF Norway wants to commit some of their budget once more for a next phase. We are planning for a sprint in the end of January 2015, and hopefully an additional one somewhere in March.

Jaap Jansma and me are now at the Edale CiviCRM sprint doing some of the ground work of what will be the extension CiviRules. Last year we developed like mad, this time we really want to have better preparation and think before we act :-) So we have been working on a framework and mockups. We are quite excited about what we have done so far, which is based on the Drupal Rules module. We decided not to 'simply' do some integration with the Rules module because that would mean only fundraising organisations using Drupal could benefit from our work. We think it is going to be important stuff for all fundraisers (and indeed for the whole community of CiviCRM users), so we want to develop this as a CMS independent CiviCRM extension.

CiviRules will have a combination of Events, Conditions and Actions. We invite you all to have a look at our wiki page http://wiki.civicrm.org/confluence/display/CRM/CiviRules+extension . We would love feedback, and really hope that some other organisations will be prepared to fund some of the development too! If you are, please drop us a mail at erik.hommel@civicoop.org.

Pogstone has published a new native extension called "Payment Processor Helper" Its designed to standardize and streamline, and ease support challenges related to automated recurring contributions that are connected to payment processors. This extension is NOT a payment processor. Its designed to work along side existing payment processors, which at this point includes PayPalPro, Authorize.net and eWay recurring.  Hopefully this is of use to organizations using one of those 3 payment processors, and/or other developers creating payment processors that want to offer automated recurring contributions.

If you are currently an organization using Authorize.net, PayPal, or eWay, this extension will provide more robust processing of automated recurring contributions from the payment processor. Plus you can use the custom search to look at the history of IPN/silent post communications.

If you are a developer working on a new payment processor, you may want to consider enhancing the scheduled job and the custom search in this extension to handle the messages/data from your new processor.

You can download and learn more about this "Payment Processor Helper" extension at: https://civicrm.org/extensions/payment-processor-helper

-Sarah Gladstone

http://pogstone.com

CiviCRM has had an array of powerful import tools for as long as I can remember but the availble options have been limited. Contacts, Contributions, Participants, Memberships and Activities...sure....but what about the Events themselves? Sorry. What about Grants? Nope.

Enter the API CSV import GUI extension, created by awesome community member Eileen McNaughton! That's a whole lot of acronyms, but basically what it allows you to do is import potentially any entity into Civi. Grants? Yes. Events? Yes. Cases? Yes! Pledges? Why not?

Translations: For those that don't know, CSV is a common, simplified spreadsheet format that can be opened using Excel, Libre Office, etc. API is some programmer-speak and GUI just means you can point and click.

I recently used this tool to import over 4,000 Grants for McKenzie River Gathering, a community supported foundation providing nearly half a million dollars a year to support innovative organizing work in Oregon addressing racism, poverty, inequity, and environmental destruction.

Simply put, this powerful new tool can turn a backlog of complex spreadsheets into a fully realized CiviCRM more easily than ever before. If you have some old data lying around the office and a mind for the technical, read on...otherwise contact a helpful provider to give you a hand.

Here's some technical tips from my experience that may be helpful.

1. Make backups before, during and after any import process

2. Learn how to install extensions

3. Some entity fields are required during import, so do some "test runs" to learn which

4. Sometimes you need to import the field value or id, not the label (i.e. "2" instead of "Pending")

5. You can supplement your efforts with clever queries using PHPMyAdmin

6. This CSV GUI import tool can be used to insert new data as well as update existing data

7. If you need to import specific id numbers rather than rely on Civi's auto-increment, import blank data first with id# and bogus data such as "Grant Type" = 0.  Then backfill this data later with and update and delete the unused rows that are still blank.

Volunteers play an important role in nonprofit organizations not only in terms of the time they donate, but also because they serve as ambassadors for organizations and frequently become regular donors.  It takes time, effort and good tech tools to grow and manage their volunteer programs effectively. Recruiting, assuring training and skills, sending reminders and giving ongoing feedback are all important to having happy volunteers and a successful project.

Even if you don’t have a formal volunteer program in your organization, like many community organizations, you probably have hard workers who are contributing regularly to your operations. Or maybe you have periodic or annual events that need staffing. CiviCRM enables you to manage information about your volunteers and ensures you have timely and appropriate interactions.

Hopefully you have heard by now of the CiviCRM Extension called CiviVolunteer. The goal of CiviVolunteer is to use CiviCRM’s already powerful tools for managing relationships and tailor them to the purpose and process of managing a volunteer program, including your scheduling needs.

We would like to invite you to play a key role in helping to shape the development of CiviVolunteer.  Please share your experience by participating in our survey. This is one of the benefits of open source, you have the opportunity to influence the direction and scope of CiviVolunteer. Everyone who fills out the survey will be eligible to win one of several $25 Amazon gift certificates.

Click here to take the survey. Please complete the survey by December 21, 2014. 

Whether your nonprofit or social good organization is small, like the local animal shelter, or large, like the Creative South conference, chances are volunteers mean a lot to your mission. Ginkgo Street Labs has been leading the effort to develop the CiviVolunteer extension of CiviCRM to make it easier and faster for nonprofits to manage their volunteers.

Because of the CiviCRM project’s open-source structure, many nonprofit users are active on the online forum, troubleshooting issues and sharing ideas for things they wish CiviCRM could do. We regularly organize and categorize this input into the CiviVolunteer Road Map, which shows where we’re going next with development.

Right now, we are in the final stages of developing CiviVolunteer Version 1.4, which includes:

• An easy way to thank volunteers for a job well done—improving their interactions with your organization

• Forms that allow managers to capture volunteers’ general interest, skills and availability

• Give managers an easy way to create multiple volunteer sign-up forms that display differing custom fields (important for nonprofits with diverse volunteer needs)

• And more…

 Of course, there’s always a next step, and we already have a list of features to be developed for release 1.5. As Ginkgo Street Labs continues to raise funding for development, we will be building out the next extension to give volunteer managers the power to:

 Streamline scheduling. You will be able to save draft schedules, set regularly recurring volunteers and send automatic notifications when volunteers are scheduled.

• Manage large groups of volunteers easily. You will be able to assign volunteers to a supervisor and set a primary location for them to report to, as well as create incentives to reward your best volunteers.

• Know more about your volunteers. You’ll gain the ability to set prerequisites (such as “must be CPR certified”) and you’ll get notifications if your volunteers don’t fit them. You’ll also have increased power to track and view data about a volunteers through the Contact Record.

• Create custom event templates and easily duplicate events.

• And more…

With each iteration, the Volunteer extension serves the needs of a growing variety of organizations. Visit the forum to join the conversation, and follow Ginkgo Street Labs on Twitter to find out when new releases are coming out. Help shape the future development of CiviVolunteer by sharing your feedback in this survey by Dec. 21, and enter to win a $25 Amazon gift certificate.

When preparing an email newsletter, one part of it that is time consuming is gathering together all the content that is needed. In my experience, virtually all the content already exists elsewhere, such as in the local CMS, in CiviCRM, or on a blog, or some other online source.    So I was thinking how can I make this process easier.  What I did: I created mail merge tokens for CiviCRM that autofill a list of recent blog posts, stories, or any other type of CMS content.  So the end-user sees a list of tokens, one for each content type and for each date range. Such as "Content of type 'blog' changed in the last 7 days" .  What is particulary powerful about this approach, is that if you are also using a CMS aggregator (such as the aggregator module in Drupal core) then virually any external RSS feed is turned into CMS content, which is now available as a CiviCRM token. 

Some examples of how this new extension may help your organization:

   - Your staff posts new content of type "story" each week.Your monthly newsletter editor can use the new token for "Content of type 'story' changed in the last 1 month" to save time preparing the newsletter.

  - A national organization you are affiliated with has a number of blogs that they host. Your local organization would like to include recent blog posts from the national organization in the local member newsletter.  Your local webmaster previously configured the aggregator module to pull in those external blogs into your CMS. Your monthly newsletter editor can use the new token for "Content of type 'feed item' changed in the last 1 month" to save time preparing the newsletter.

- Any other situation where there is existing content that you want to include in your email or PDF.

This new extension "Content Tokens" is published in the CiviCRM extensions area https://civicrm.org/extensions/content-tokens 

This new extension is designed in the same style as the "Fancy Token" extension that provides tokens for upcoming events, contribution pages, profiles, and WebForms. 

-Sarah     sarah@pogstone.com 

At Web Access we’re constantly exploring new technologies and challenging our engineers to come up with new, and useful software solutions. This exercise allows us to hone our skills, motivate our teams, and keeps us thinking outside of the box. At times, such as this, it gives us a little something to boast about.

We thought it would be really useful for the community if there were a quick way to donate in CiviCRM. So we set out to simplify the donation process. What we’ve come up with is the “Quick Donate” extension which allows the administrator to link existing donation pages with our new sleek, user friendly and responsive donation page. The genesis of this idea came from the blog post here

Pen and paper ready? Here’s how it works…

1. Link your existing contribution page with quick donate form. (Menu: Administer >> CiviContribute >> Quick Donate Configuration)
2. Well, that’s it really… now you can use your new quick donation form (Menu: Contribution >> Quick Donate >> Test mode and Live mode)

The Quick donate form is responsive and works well across devices with various resolution and sizes. This is a single page form with 3 distinct sections.

Step 1: Select the amount

Steps 2 & 3: Enter donor and credit card information

On successful donation the donor will be redirected to simple thank you page:

We have tested the quick donation process with PayPal Pro and Authorize.net but it should work (in theory) with any payment processors where credit card information is entered upfront.

Currently, you can find the code at on GitHub and we will be submitting this as an extension shortly.
 

The ability to create petitions in CiviCRM was a tremendous move forward for people using CiviCRM for political organizing. The petitions feature took another step forward with the Petition Email extension, that allows copies of your petition to be automatically sent by email to a given target. However, the holy grail of e-advocacy was still just out of reach: automatically sending your petition to the petition signer's elected official.

Now, CiviCRM will have it.

The Progressive Technology Project is putting our final touches on a ground-up re-write of the Petition Email extension that will allow the target to be chosen dynamically based on the information provided by the petition signer. You can still specify one or more people to receive a copy of all petitions signed. However, with the re-write, you will also be able to specify a group of contacts that will get the petition if a defined set of fields match the petition signer.

The extension will not do the address to electoral district matching for you. To fully implement this feature, you will need another extension to handle the lookups (for example, we are using the Drupal CiviCRM Cicero module). Any third party module or extension that automatically update each contact after it is saved with the right district information will work with Petition Email (and it seems that Google is making an API for this kind of lookups available for free if anyone is looking for a good extension project).

How does it work? 

First, setup your module or extension for doing address to district lookups (using, for example, the Drupal CiviCRM Cicero module). You will need to create custom fields to hold your election district information (e.g. state senate district id, city council district id, etc). Make sure that new contacts are getting properly coded to the right districts.

Next, create one or more target groups in your database. For example, you might import all of your state senators or your state assembly people or city council members. Be sure that these contacts are properly coded to their districts (using the same custom fields).

Then, create your petition. You will see an extra checkbox called "Send email to target." Clicking this checkbox shows a number of new fields which allow you to specify how the email messages should be sent.

The critical section allows you to specify a set of static targets and up to three dynamic target groups with matching fields. For example, if you are running a state-wide campaign, you might specify the Governor and Lieutenant Governor as static targets, since they represent everyone in the state and should get a copy of all petitions. You then could specify the state senator group as one target (which would be paired with the state senate district id) and the state assembly group as the second target (which would be paired with the state assembly district id).

When a petition signer puts in their address, their senate and assembly district ids would be looked up and recorded. Then, the Petition Email extension would find any contact in the state assembly district with the same value for the state assembly district id and any contact in the state senate district with a matching state senate district id. These contacts, along with the static contacts, get a copy of the petition.

Want to try?

We haven't yet released it formally as an extension because it is still being tested. But that doesn't mean you can't give it a spin. In fact, we'd like that very much. It currently lives in the dynamic-targets branch of our git repo. If you find a problem, please let us know by opening an issue.

In 2014 MAF Norge funded CiviCooP developing the TriggerAction extension (https://github.com/CiviCooP/org.civicoop.triggers) to assist them with automated Donor Journeys in CiviCRM. Our team of Jaap & Erik from CiviCooP and Steinar & Helen from MAF Norge worked hard for a week and we came up with something that does work but is fairly complicated and decidedly unsexy. During the CiviCRM sprint in Edale Jaap & Erik from CiviCooP worked on a first analysis of CiviRules, a CiviCRM extension that is based on the Drupal Rules module. We described our thoughts (and some great contributions from community members) on a wiki page: http://wiki.civicrm.org/confluence/display/CRM/CiviRules+extension

In 2015 our aim is to actually develop and deliver the CiviRules extension if we get enough funding from the community. So we are really happy to mention that MAF Norge has agreed to fund another 10 days of development in 2015, so we can make a start with a sprint in January 2015. In this sprint we will add some specific stuff to the TriggerAction extension for MAF Norge's immediate needs, and start with the first development of CiviRules as we both believe that should replace the TriggerAction stuff.

Even better news: Amnesty International Flanders have promised us to fund some development too. On top of this the fundraising consultant Ilja de Coster will also contribute, and we as CiviCooP will use some of our 2014 profit to fund some development too. Enough to be happy to announce that we will be planning a second sprint, solely focused on the further development of CiviRules.

We are aiming for the week of 9 March 2015. We do not expect to be near a first completion with 2 sprints, so we hope to get some more funding during the year. if your organisation, having read the documentation on what we think CiviRules should do, is interested in funding please drop either me (erik.hommel@civicoop.org) or Martin at MAF Norge (martin@maf.no) a line! If you are a developer and would be interested in taking part in this second sprint (probably week 9 March 2015) please let us know too!

I recently had a requirement to allow for export of Accounting Batches to a format that AccountEdge can use. The good news was AccountEdge has many CSV import wizards for many different types of data. The bad news was the layout of the accounting batch export CSV file from CiviCRM did not match what AccountEdge expected.  So I set about creating a new export format for CiviCRM which is compatible with the "File ... Import Data ... General Journal Entries" wizard in AccountEdge.

What I had to do:

Step 1:  Update list of export options to include the new choice "CSV for AccountEdge"

During development, I had to export the same batch several times to verify my code was working the way I wanted.  Normally in a production environment, its impossible to re-export the same batch again.  So what I did to re-export:

1) Using the CiviCRM back office, I deleted the activity "exported accounting batch" that was associated with my batch

2) Using SQL, I changed the batch status back to "closed"  my SQL:  " Update civicrm_batch  set  batch_status_id = 2  where id = xxx " 

Some thoughts on how this could be improved:

My first (failed) attempt to do this was to create a new CiviCRM extension using civix, since I figured other people are using AccountEdge and may want this new export format.   However, I discovered that an extension will not work for this situation, and I had to change core files. (I tried overriding the core files in my extension, but then the classloading failed for the core export formats)

Some steps that could improve this:

1) The options for "export formats" should be stored in the database.  This should include the mapping to which class implements each option. (Like what is already in place for custom report templates)  Ideally, one choice could be set as default, or other choices disabled (If an organization is using AccountEdge, its confusing to the bookkeeper to be defaulted to IIF when exporting batches) 

2) Allow the implementation classes to be packaged in an extension. Ideally this would mean a new "mgd" file that allows the extension author to declare the new export option and label, and not have to manually update the database where the options are stored. (ie like what is already possible in Extensions for report templates) 

-Sarah Gladstone 

sarah@pogstone.com

As mentioned by Tony in his blog, we have recently released “Quick Donate” extension that allows you to have a pretty and responsive donation pages. I want to take this opportunity to share some implementation details for this extension.

CiviCRM and API

We made a conscious effort not to add more configurations and decided to use existing contribution page settings. So the only thing that you need to do is link your existing contribution page with our extension.

AngularJS

CiviCRM already has support for angular extension. It was fairly easy to learn and develop front-end based on angular framework. It also helped in implementing client-side validations for most of the user related and payment related fields. Angular was useful to script out the single-input credit card field, which provides a more concise input method for entering credit card info.

Bootstrap CSS

One of the aim of this project was to make donation pages responsive and it should work across various devices. Hence we decided to use Bootstrap CSS.

Bootstrap is one of the most popular HTML, CSS, and JS frameworks for developing responsive, mobile first projects on the web. Bootstrap's responsive CSS adjusts to Desktops, Tablets, and Mobile phones.

Ziptastic library

We wanted to make donation page more user centric and reduce the number of obstacles in donation. Our extension uses Ziptatstic postal code library to auto-populate “City” and “State” information based on the zip code. Since this is US centric service, we also provide an option to disable this service in extension configuration.

Protractor Test Runner

Tests are an integral part of any project. Our front-end is based on angular hence we decided to use Protractor. Protractor is a Node.js program, and runs end-to-end tests that are also written in JavaScript and run with node. Protractor uses WebDriver/Selenium Server to control browsers and simulate user actions. More information can be found at https://docs.angularjs.org/guide/e2e-testing.
In this extension we are using CiviCRM selenium server configuration, to run protractor test cases.

Currently we have 37 tests and some of the use cases are listed below:

• Redirection to correct page or display error if case of incorrect configuration
• Quick donate configuration
• Conditional display of “State” and “Country” based on "Ziptastic" configuration
• Validations for credit card type, number, expiration date
• Required field validation
• Ensure contributions are created correctly for existing as well as anonymous user

You can find the code at https://github.com/webaccess/com.webaccessglobal.quickdonate
 

My first exposure to CiviCRM was around 2009. I was working at Freeform Solutions, Canadian IT nonprofit organization that supports other nonprofits by providing technology support. We were looking for a CRM system that would support an HR organization that had a few thousands contact records. They needed a software that would allow all staff to access the same contacts database so that all staff members can update the records, send newsletters and other mass mailings and to run reports on the collected data to know what type of clients they work with. In the future, they were planning to do online events registrations, possibly with payments.
We've done some research and several things attracted us to use CiviCRM for this project:
- CiviCRM, being an open source software, meant to us, the developers, that if we had to tweak any functionality or add some new features, we could do that without having to rely on the creators of the software to allow us to do it or have to pay them to do it
- CiviCRM would store all the information in one database and the information from event registrations would be seen on individual contact records which was a much better solution than using several systems to do bits of this implementation
- CiviCRM had all the components that we were looking for
- CiviCRM, being a web-based application, allowed all staff to have access to the data, not only on any computer in the office but also on the go, when travelling or working from other offices. There was no additional costs no matter how many users were logging and using the system
- CiviCRM being an easy to expand and customize software - by adding custom fields - meant that there was no large additional cost to adjust that software to the organization's individual need - to collect organization specific data. In addition, designated staff could add those custom fields using the CiviCRM interface (point and click, no need for specialized programming knowledge) which meant lower development costs in the future.

That project marked the start of my journey with CiviCRM. I attended one of the first CiviCon in San Fransisco in 2010 and although we were a small group at that time (about 100 people?), it was fascinating to see what others have done with CiviCRM, what projects had been implemented with amazing results.

Since then, while working at Freeform Solutions, I've implemented or managed an implementation of a few hundred CiviCRM projects, ranging from small implementations to support staff shared database of contacts to large implementations where either almost every single CiviCRM feature was turned on or where we took CiviCRM core and extended one feature to be the hub of the whole system.
Almost every implementation had used membership management, online event registration, online and offline payments, tax receipts, mass mailing, reports and in a case where we extended the core functionality, we've implementation for Edmonton's Food Bank in 2012 a food hamper management system built on CiviCRM Activities. In recent years, Freeform Solutions expanded and was able to contribute back to CiviCRM community thanks to Lola Slade's coding expertise and Lynna Landstreet's great documentation contributions.

For the last 5 years, I had a pleasure to be both a web developer and a project manager for CiviCRM projects. My favourite part is the discovery phase, where together with a client, we discuss the system needs and plan how CiviCRM will be used to meet those needs. Because CiviCRM has many data management parts, it is important to plan the implementation to take the advantage of all the built-in features. Incorrect implementation (true to any project process) often causes misplacement of the data and in consequence, not being able to take the full advantage of CiviCRM features. I have seen implementations where organizations would create custom fields for email, contacts names etc. without knowing that those fields are provided out of the box. In this case, using custom fields for storing email address or contact names could render those fields unusable for mass mailing or keeping the database clean (so called dedupe process which we wrote about with Lynna). Proper design and planning of implementation of any project allows to lower the final implementation costs and future maintenance costs, so it is a worthy investment.

The good news is that, once the planning is completed, often to the client surprise, their needs can be met by simply configuring existing CiviCRM out-of-the box functionality. Sometimes, small tweaks are needed and those are usually either met by creating custom code or using one of the existing CiviCRM extensions that other developers so generously provide to the community. CiviCRM Extensions directory was one of the community projects that I helped to implement, to enable everyone to find all the great CiviCRM extensions in one place.

To better align my passion with client needs, in January 2015 I moved on to my own consulting company Kasuwade Solutions Inc. where I will continue to work with open source systems to address organization's IT needs. As CiviCRM is my favourite CRM, I will certainly continue to be part of this community, using my knowledge and years of experience to support sucesfull CiviCRM implementation.

For any of you, I encourage you to join CiviCRM community:
 
- spread the word about CiviCRM by registering your CiviCRM installation,

- become a CiviCRM partner (if you are a development shop) or become a CiviCRM member (if you are a CiviCRM user) - to help support growth of CiviCRM, which means a better tool for us all to work with and use.

In the UK we’re used to being able to lookup our addresses based on our postcodes and charities add this to their wish lists for their own sites. However, once they establish the costs associated with this, they often find the ROI isn’t in the black and drop the idea.

There’s some good news………

Since July of 2013 Royal Mail unveiled its improved access to Postcode Address file including “Free access to PAF for independent small charitable organisations” i.e. those that have less than £10m per annum income and who are registered charities or CICs. So small charities can now get access to the PAF file for free, which is great, what we are proposing is an extension to allow charities to import the PAF file into CiviCRM (including Updates) and enable Postcode Lookups on Address fields.

There are of course other solutions out there but they’re paid for services and have obviously pushing out words of caution about using the PAF files. However, the power of open source and crowd funding should ensure that we can produce a solution that is not only free to use but functionally able to match, if not better, the paid solutions. We currently have an extension that allows users to lookup addresses using AFD postcode or Experian (you can see an example here) and we’re proposing to enhance this extension to introduce the core PAF files as an option, so you’ll be able to provide your supporters and staff the ability to use Postcode Lookups for free!

Now for the sell……

We need to make the following amendments to the extension;

• Produce an interface to allow for the upload PAF files, including updates and verfication of file format

• Amend the postcode lookup to use raw PAF files

• Amend to allow both public and authenticated use

• Ensure the extension is easy to deploy

If you’d be interested in hearing more or would like to get involved in one form or another please post your comments here. We’ll push out a wiki page and a MIH in due course…..

Thanks and Merry Xmas from the Veda NFP Consulting Team......

We have just created an extension for MAF Norge that produces an XML file for the tax authorities with the deductible amount for the year for each donor. With a little configuration customization (detailed in the documentation on GitHub) it can be used by any Norwegian organization that has to produce this file every year. MAF Norge is currently testing the extension. It can be found in the extension list on this website: https://civicrm.org/extensions/yearly-tax-declaration-donors-norwegian-xml-format and on Github: https://github.com/CiviCooP/no.maf.oppgavexml

Thanks to MAF Norge for funding this development!

Extension can be used and modified as you like, we will only maintain the requirements from MAF Norge at this point in time.

This week we have been working for MAF Norway to automate the sending of SMS, E-mail and PDF creation. What we have done in the past for them is that we created a Trigger/Action module to automate their donor journeys. The actions in the trigger/action module are API calls. But what was missing up-to-today was to functionality to send e-mail, or to create a PDF Letter or send an SMS by invoking an API Call.

Note: the trigger/action module is a very experimental module just developed for the MAF. At the same time we are developing a CiviRules modules which will work in a similair way as the Drupal Rules module.

This week we developed three extensions to accomplish this:

• org.civicoop.smsapi
• org.civicoop.emailapi
• org.civicoop.pdfapi

org.civicoop.smapi

The SMS Api has one action Send this action takes three parameters:

• contact_id - the id of the contact to which you want to send the sms
• provider_id - the id of the SMS provider which you want to use to send the sms
• template_id - the message template idof the message you want use

org.civicoop.emailapi

The E-mail Api has one action Send this action takes two required parameters:

• contact_id - the id of the contact to which you want to send the e-mail
• template_id - the message template idof the message you want use

There are also two optional parameters:

• from_name - the name of the sender
• from_email - the e-mail adress of the sender

org.civicoop.pdfapi

The PDF Api works a little bit different from the SMS and the E-mail API. Because the PDF files have to be stored at a location where somebody could print them. We have decided to use an e-mailaddress for this because that was the easiest way in processing the print job.

The PDF Api has one action Create this action takes three parameters:

• contact_id - the id of the contact to which you want to send the sms
• template_id - the message template idof the message you want use
• to_email - the e-mail address which will receive the pdf file. This could be the e-mail address of your printer or of someone who will do the printing of the pdf letters